Tagged articles
GPU memory
29 articles · Page 1 of 1
This article walks through vLLM’s token‑generation pipeline by dissecting its source files, process roles, ZMQ communication, scheduling, and deployment considerations, revealing how prompts become streamed token IDs and highlighting the key factors that limit concurrency and performance.
GPU memoryLLM inferenceZMQ
0 likes · 17 min read

The article explains why large‑model inference services frequently hit GPU memory limits, breaks down static vs. dynamic memory consumption, shows how KV‑Cache, request length, and concurrency amplify usage, and provides a step‑by‑step troubleshooting and mitigation workflow for production environments.
GPU memoryInference OptimizationKV Cache
0 likes · 26 min read

Although a KV‑cache eviction strategy can discard 90% of tokens, GPU memory usage stays almost unchanged because paged‑attention memory blocks remain occupied and fast attention kernels discard the full score matrix, preventing effective memory release.
FlashAttentionGPU memoryKV Cache
0 likes · 7 min read

Training 100‑billion‑parameter Transformers is limited by GPU memory rather than compute, requiring a mix of tensor parallelism within nodes and data parallelism across nodes, along with pipeline parallelism, gradient accumulation, and careful framework choices to balance memory, bandwidth, and compute overheads.
GPU memoryPipeline Parallelismdata parallelism
0 likes · 14 min read

This article provides a practical, data‑driven comparison of Full Fine‑Tuning, LoRA, and QLoRA for adapting 7B‑70B open‑source LLMs, detailing memory requirements, training speed, cost, performance trade‑offs, step‑by‑step workflows, code examples, evaluation metrics, common pitfalls, and optimization tips to help engineers choose the most suitable fine‑tuning approach for their data and budget.
Full Fine-tuningGPU memoryLoRA
0 likes · 24 min read

The article explains the seven sources of GPU memory usage during PyTorch training, shows how to measure them with built‑in profiling APIs and the memory‑viz tool, and evaluates three effective optimizations—gradient checkpointing, mixed‑precision training, and optimizer selection—detailing their memory savings and performance costs.
GPU memoryGradient CheckpointingPyTorch
0 likes · 8 min read

This article analyzes why vLLM's PagedAttention can cause GPU memory fragmentation and out‑of‑memory errors in production, presents four typical OOM scenarios, and provides concrete diagnostics, configuration tweaks, code examples, and monitoring strategies to eliminate the problem.
CUDAGPU memoryLLM serving
0 likes · 22 min read

The article dissects DeepSeek‑V4’s local deployment using vLLM, explaining the steep hardware requirements, the complex heterogeneous KV‑cache architecture, and the aggressive kernel‑fusion and multi‑stream optimizations that together make high‑context inference both memory‑intensive and engineering‑heavy.
GPU memoryKV Cacheattention compression
0 likes · 15 min read


Qborfy AIMar 24, 2026 · Artificial Intelligence
This article explains full fine‑tuning—updating all parameters of a pretrained model—to achieve the highest task performance, compares it with LoRA and prompt tuning, shows when it is appropriate, provides a step‑by‑step Hugging Face implementation, memory‑saving tricks, common pitfalls, and practical takeaways.
DeepSpeedFull Fine-tuningGPU memory
0 likes · 9 min read

This guide explains the five common sources of GPU memory consumption in large‑model inference services, provides a step‑by‑step diagnosis workflow—from static usage and KV‑Cache analysis to concurrency and K8s scheduling—offers concrete command‑line checks, scripts, configuration examples, and actionable remediation and monitoring recommendations.
GPU memoryKV CacheLLM OOM
0 likes · 28 min read

The article dissects DeepSeek’s new Engram architecture, which separates computation from memory by using a large, cheap‑RAM‑based lookup table to store factual knowledge, allowing the transformer’s compute layers to focus on reasoning, dramatically reducing GPU memory demand while improving code, math, and long‑context performance.
EngramGPU memoryMemory-Compute Architecture
0 likes · 7 min read

This report analyzes the memory bottleneck of DeepSeek‑V3.2‑Exp’s sparse‑attention decoder, proposes the Expanded Sparse Server (ESS) to offload the latent cache to CPU memory, and demonstrates through high‑fidelity simulation that the approach dramatically improves decode throughput while keeping latency within acceptable limits.
Cache offloadGPU memoryLLM inference
0 likes · 20 min read

This article breaks down the GPU memory requirements of large language models during training and inference, detailing the contributions of model weights, optimizer states, activations, KV cache, and activation recomputation, and provides concrete formulas, examples, and scaling insights for models like Qwen3 and DeepSeek V3.
GPU memoryKV CacheLLM
0 likes · 18 min read

The article explains how to estimate GPU memory and overall hardware requirements for fine-tuning and training large dense and MoE models, detailing calculations for full-parameter and LoRA approaches, scaling rules, and hidden costs relevant to interview assessments.
GPU memoryLoRAfine-tuning
0 likes · 8 min read

This article explains how to estimate the GPU memory required for running large language models of 3 B, 7 B, and 13 B parameters, walks through step‑by‑step calculations, shows how hardware limits affect feasibility, and offers practical optimization techniques such as quantization and CPU offloading.
AI model sizingCPU OffloadingFP16
0 likes · 5 min read


ArchitectMay 18, 2025 · Artificial Intelligence
This article breaks down GPU memory consumption for large Transformer models, explains how to estimate each component—parameters, optimizer state, activations, gradients—and shows how parallelism, mixed precision, and recomputation strategies can dramatically reduce the footprint.
AI TrainingGPU memoryModel Scaling
0 likes · 14 min read

This article demonstrates how to use ACK AI Profiling, built on eBPF and dynamic process injection, to perform non-intrusive, low‑overhead profiling of Kubernetes‑deployed large‑language‑model inference services, identify GPU memory growth causes, and apply optimization recommendations to prevent OOM issues.
AI profilingGPU memoryKubernetes
0 likes · 10 min read

After deploying the full‑precision DeepSeek‑R1 model on a 2×8‑GPU ACS cluster, repeated stress tests showed GPU memory usage continuously rising without release; this article details the investigation, reproduces the behavior, examines vLLM logs, Prometheus metrics, and reveals PyTorch’s caching allocator as the root cause, offering mitigation tips.
DeepSeekGPU memoryMemory Cache
0 likes · 21 min read

This article walks through a suite of PyTorch techniques—including automatic mixed precision, BF16, gradient checkpointing, gradient accumulation, tensor sharding, efficient data loading, in‑place ops, lightweight optimizers, memory profiling, TorchScript, and kernel fusion—that together can cut peak GPU memory usage by up to twenty‑fold while preserving model accuracy.
GPU memoryGradient CheckpointingOptimizer
0 likes · 13 min read

This article explains a simple formula for estimating the GPU VRAM required to serve large language models, demonstrates the calculation with a 7‑billion‑parameter example, clarifies why a 20% safety buffer is needed, and offers practical strategies such as quantization, CPU offload, and multi‑GPU parallelism to reduce memory usage.
GPU memoryLLMVRAM estimation
0 likes · 6 min read

The article investigates why Nvidia L40S and RTX A6000 GPUs show less memory via nvidia‑smi than their advertised 48 GB, revealing that enabled ECC memory reserves a few gigabytes, and demonstrates the effect by toggling ECC on a Tesla‑T4 card.
ECCGPU memoryL40S
0 likes · 4 min read

This article explains the technical differences between GDDR and HBM GPU memory, compares their bandwidth, cost, and use‑case scenarios, and helps engineers decide which memory type best fits their performance and efficiency requirements.
GDDRGPU memoryGraphics
0 likes · 8 min read

This article provides a comprehensive, step‑by‑step guide to training customized large language models, covering industry‑specific needs, data privacy, meticulous data cleaning, optimal data‑ratio balancing, token budgeting, GPU memory accounting, LoRA fine‑tuning techniques, and practical evaluation metrics for robust AI deployment.
AI TrainingGPU memoryLoRA
0 likes · 23 min read

This article explains how GPU memory is allocated during model fine‑tuning, describes collective communication primitives, and compares data parallel, model parallel, ZeRO, pipeline parallel, mixed‑precision, and checkpointing techniques for reducing memory consumption in large‑scale AI training.
GPU memoryPipeline ParallelZeRO
0 likes · 9 min read

This article explains how vLLM’s PagedAttention, inspired by operating‑system virtual‑memory paging, dynamically allocates KV‑cache memory to dramatically reduce GPU memory fragmentation, improve throughput, and handle scheduling, preemption, and distributed inference for large language models.
GPU memoryLLM inferencePagedAttention
0 likes · 25 min read


NewBeeNLPFeb 5, 2024 · Artificial Intelligence
HiFT introduces a layer‑wise hierarchical fine‑tuning strategy that freezes most parameters per step, reduces optimizer state memory, and adapts mixed‑precision training, enabling 7B and 13B models to be fine‑tuned on 16‑31 GB GPUs while maintaining competitive performance.
GPU memoryHiFTLLM fine-tuning
0 likes · 12 min read

This article explains the challenges of large‑model offline (batch) inference, such as GPU memory limits and distributed scheduling, and shows how Ray’s cloud‑native architecture, model partitioning, and Ray Datasets can be used to build efficient, elastic inference frameworks deployed with KubeRay.
GPU memoryRaydistributed computing
0 likes · 18 min read

This article introduces OneFlow Coop, a memory‑optimization technique that jointly optimizes dynamic‑graph recomputation strategies and GPU memory allocation by analyzing existing DTR limitations, proposing recomputable in‑place, op‑guided tensor allocation, and layout‑aware eviction modules, and demonstrating superior experimental results.
Dynamic GraphGPU memoryOneFlow
0 likes · 18 min read

Android apps often get killed due to high memory usage, especially from GPU caches; this article explains the Android drawing system architecture, how bitmap rendering creates GPU memory leaks, and demonstrates using WindowManagerGlobal.startTrimMemory to clear those caches while outlining common pitfalls and best practices.
AndroidGPU memoryhwui
0 likes · 12 min read
