Enterprise LLM Private Deployment: GPU, VRAM, Storage & Network Capacity Planning Guide

1. Problem Background and Common Pitfalls

The article opens with a real-world scenario: an enterprise needs to deploy a 70B-parameter model for 200 internal users with 30 peak concurrent users on an 8-GPU server. The author identifies three common misconceptions:

Pitfall 1: Ignoring KV Cache. Many only calculate static weight memory (e.g., 70B × 2 bytes = 140 GB for BF16) but KV Cache often consumes half or more of total VRAM. For a 70B model with 32K context, a single request needs ~10 GB KV Cache, potentially exceeding available memory after weights and framework overhead.

Pitfall 2: Confusing "loadable" with "usable". Successful vllm serve startup only proves weights fit. Real metrics are GPU KV cache size and Maximum concurrency for xxx tokens per request from startup logs.

Pitfall 3: Overlooking storage and network bottlenecks. Loading 140 GB over 1 GbE takes ~22 minutes; concurrent instance restarts can saturate storage networks. East-west traffic (tensor-parallel all-reduce) is the real network bottleneck, not north-south client traffic.

2. Applicable Scenarios

The methodology applies to: (A) new inference cluster sizing, (B) existing cluster expansion evaluation, (C) model upgrade feasibility checks, (D) domestic accelerator (Ascend 910B, Hygon DCU) adaptation, and (E) cost optimization reviews. It does NOT cover training/full fine-tuning capacity planning, pure API gateways, or single-GPU experiments.

3. Core Knowledge: VRAM, Storage, Network Breakdown

3.1 Six VRAM Components

Model Weights (static): Parameters × bytes per parameter. Table shows sizes for 7B/14B/32B/70B at FP32, BF16/FP16, INT8/FP8, INT4 (GPTQ/AWQ). INT4 actual overhead ~0.56-0.58 bytes/param due to group-wise quantization scales.

KV Cache (dynamic): Grows linearly with concurrency and context length. Core formula: 2 × layers × KV_heads × head_dim × dtype_bytes per token.

Activations/Workspace: Prefill-phase peak depends on max_num_batched_tokens. For 70B at 8192 tokens, ~3-8 GB per GPU.

Framework/Runtime Overhead: CUDA context (300-600 MiB), PyTorch caching allocator, NCCL buffers, logits materialization (e.g., 128K vocab × 256 batch × 4 bytes = 128 MB). Reserve 1.5-2.5 GB per GPU.

Driver/Display: Consumer GPUs (RTX 4090) use 300 MiB-1 GB for display; datacenter GPUs minimal.

Fragmentation/Pre-allocation Margin: Controlled by gpu_memory_utilization (default 0.90).

3.2 Weight VRAM Calculation

Prefer reading actual safetensors file sizes via

du -sb *.safetensors | awk '{s+=$1} END {printf "%.2f GiB
", s/1024/1024/1024}'

. Safetensors maps zero-copy; PyTorch .bin may double peak memory during load.

3.3 KV Cache Derivation

Formula: 2 × layers × KV_heads × head_dim × 2 (FP16) bytes per token. Examples:

Llama-3.1-70B (80 layers, 8 KV heads, head_dim=128): 320 KiB/token → 9.77 GiB for 32K context.

Llama-3.1-8B (32 layers, 8 KV heads): 128 KiB/token → 4 GiB for 32K.

DeepSeek-V3 (MLA): 61 layers, kv_lora_rank=512, qk_rope_head_dim=64 → 68.6 KiB/token, 4-5× smaller than GQA.

Concurrency estimation:

Available_KV_VRAM = Total_VRAM × gpu_memory_utilization - Weights - Activations - Framework

;

Max_Concurrency = (Available_KV_VRAM / per_token_KV) / P95_Context_Length

.

3.4 GPU Selection Criteria

Table compares V100, A100 40/80G, A800, H100, H800, L40S, L20, RTX 4090, A6000, Ascend 910B. Four decision points:

Capacity vs Bandwidth: Decode is memory-bandwidth bound. Theoretical tokens/s ≈ Bandwidth / Weight_Bytes. A100 80G (2039 GB/s) → ~14.6 tok/s for 70B BF16; L40S (864 GB/s) → ~6.2 tok/s.

NVLink Necessity: Tensor-parallel all-reduce per layer. TP≤2 on PCIe 4.0 acceptable; TP≥4 requires NVLink. nvidia-smi topo -m shows PHB (cross host bridge) vs PIX (same switch) latency differences.

ECC & Compliance: RTX 4090 lacks ECC; not recommended for production.

Precision Support: FP8 needs SM89 (L40S/4090) or SM90 (H100); A100 (SM80) lacks native FP8. Domestic accelerators need separate operator/quantization validation.

3.5 Storage: Four Roles

Model Repository: Read-heavy, large sequential reads. 70B BF16 ~140 GB; keep 3 versions → ~500 GB. Bandwidth critical: 1 GbE = 22 min load; 25GbE = 50 sec; local NVMe = 25-45 sec. Plan for concurrent multi-instance restarts.

KV Cache Offload/Prefix Cache (optional): 8K context prefix ≈ 2.44 GiB (70B). 1000 entries = 2.4 TB. Defer until hit-rate data exists.

Logs/Audit: 10k req/day × 2 KB = 20 MB/day (summary); full prompt/completion ~200 MB/day. Retain 6 months → 36 GB. Use separate disk with rotation.

Datasets/Fine-tune Artifacts: LoRA: MBs-GBs; Full fine-tune checkpoint = model size (140 GB) + optimizer states → TB scale.

3.6 Network: Three Directions

North-South (Client↔Service): Tiny payloads (~6 KB/request). 2 QPS = 12 KB/s. Bottleneck is connection count (SSE long-lived) and proxy timeouts ( proxy_buffering off, proxy_read_timeout 300s).

East-West (Inter-GPU): TP all-reduce dominates. Prefill (8192 tokens, TP=4, 70B) = 30 GiB communication. NVLink 3.0 (500 GB/s) → 60 ms; PCIe 4.0 x16 (25 GB/s) → 1.2 s. Decode (16 concurrent) = 900 MiB/s sustained. Mixed chunked prefill averages ~75 GB/s sustained → requires NVLink/RDMA.

Storage Network: Isolate from business traffic or apply QoS ( tc HTB). Switch-side QoS preferred.

3.7 Coupling

Context length affects KV VRAM and PD-disaggregation KV transfer. Concurrency affects KV VRAM and east-west traffic. Quantization reduces weight VRAM, may enable FP8 KV, but adds dequant overhead. Model size impacts storage, load bandwidth, TP degree.

4. Five-Step Assessment Process

Step 1: Determine Model – read config.json for structural params
Step 2: Define Business Metrics – concurrency, context, TTFT, TPOT
Step 3: Calculate VRAM – Weights + KV + Activations + Framework → GPU count/type
Step 4: Plan Storage – Repository + Logs + Cache + Artifacts → Capacity & Bandwidth
Step 5: Plan Network – North-South, East-West, Storage → Bandwidth & Interconnect

5. Step 1: Read Model Config

Never rely on memory. Key fields from config.json: num_hidden_layers, hidden_size, num_attention_heads, num_key_value_heads, head_dim, vocab_size, max_position_embeddings, num_local_experts, num_experts_per_tok, kv_lora_rank, quantization_config. Python script provided to extract all. MoE models (Mixtral) need total params for VRAM, active params for compute. MLA (DeepSeek) uses different KV formula.

6. Step 2: Business Metrics Confirmation

Six metrics to agree with stakeholders:

Model & Use Case: Determines input/output length distributions (chat: 300-2k in / 200-800 out; code: 8k in / 50-200 out; summarization: 30k in / 500 out).

Concurrency Definitions: Registered users → DAU → Peak Online → Peak Concurrent Requests (≈ Online × 0.3-0.5) → Peak QPS (Concurrent / Avg Latency). Example: 40 online → 16 concurrent → 2 QPS (8s avg latency).

Context Lengths: Distinguish max_model_len (hard limit), P50, P95 (plan KV capacity), P99 (can reject/throttle).

Latency Targets: TTFT P95 ≤ 3s (chat), TPOT P95 ≤ 200ms (≥5 tok/s). Queue depth = TTFT Budget / Prefill Time.

Availability: Single-node failure tolerance? Maintenance windows? SLA 99% vs 99.9% (single replica struggles).

Growth Forecast: 6-month concurrency doubling? Model size increase? Multimodal addition (1-3 GB extra VRAM)?

Template confirmation table provided.

7. Step 3: VRAM Assessment & GPU Selection

7.1 Full Budget Formula

Per_GPU_VRAM = Weight_Shard + KV_Shard + Activation_Shard + Framework_Overhead + Safety_Margin

. Framework overhead (CUDA context, NCCL) does NOT shard with TP.

7.2 Worked Example: 70B BF16 on 8×A100 80G (TP=4, DP=2)

Weight: 145.67 GiB / 4 = 36.42 GiB/GPU

Activations: ~5 GiB/GPU

Framework: ~2 GiB/GPU

Used without KV: 43.42 GiB

KV Available: 80 × 0.90 - 43.42 = 28.58 GiB/GPU → 114.32 GiB total per instance

Per-token KV: 0.3125 MiB → 374,600 tokens total

At P95 4000 tokens: 93 concurrent (vs required 16). 5.8× headroom.

Checks: DP=2 for redundancy (99.5% SLA). P99 8000 tokens × 16 = 39 GiB < 114 GiB. 6-month doubling to 32 concurrent still safe.

7.3 If Only 4×A100 80G

Same per-instance capacity (TP=4) but no redundancy. Options: accept lower SLA, downsize to 32B (TP=2×DP=2), quantize 70B to INT4 (TP=2×DP=2), or buy another 4 GPUs.

7.4 Minimum Viable Configuration Algorithm

KV_per_GPU = GPU_VRAM × Util - Weight/TP - Activation/TP - Framework

KV_Needed = Target_Concurrency × P95_Context × Per_Token_KV

TP ≥ KV_Needed / KV_per_GPU (iterate because KV_per_GPU depends on TP)

Verify cheaper combos (e.g., L40S 48G needs TP=8 or quantization; INT4 on L40S gives 26 concurrent @ 4k but decode ~6 tok/s borderline).

7.5 MoE Special Cases

Mixtral-8x7B: Total 46.7B params (87 GiB BF16) all must reside; active 12.9B for compute. KV per token = 128 KiB (32 layers, 8 KV heads). VRAM like 70B dense, compute like 13B. DeepSeek-V3: 671B total (1342 GiB BF16) needs 17×A100 80G or 5×80G INT4; MLA KV tiny (68.6 KiB/token).

8. Step 4: Storage Assessment

8.1 Capacity Checklist

Model Repo: Σ Model_Size × 2.5-3 (current, previous, candidate) → 70B BF16 146 GiB × 3 = 438 GiB + quantized versions → ~1 TB recommended.

Container Images: vLLM ~15 GB, TensorRT-LLM ~25-30 GB, plus layers. 5 versions + cache ≈ 120 GB/node; registry 500 GB.

Logs/Audit: ~40-100 GB for 6 months full audit. Separate disk.

Metrics (Prometheus): ~1.2-2 GB/day → 110-180 GB for 90 days.

Prefix Cache (optional): Estimate after hit-rate data.

Temp Space: Max_Model × 3 = 440 GiB for downloads, quantization intermediates, GGUF conversion.

8.2 Performance Requirements

Sequential Read Bandwidth: Model_Size / Acceptable_Load_Time. 146 GiB in 3 min → 830 MiB/s (needs 10GbE+ or local NVMe).

Concurrent Startup Bandwidth: 3 instances × 830 MiB/s = 2.5 GiB/s → 25GbE+ or local cache.

IOPS: Only for KV offload; otherwise SATA SSD sufficient.

8.3 Storage Solution Comparison

Table: Local NVMe (3-7 GB/s, high IOPS, medium cost) → Recommended for model cache. Local NVMe RAID0 (10-25 GB/s) → Large multi-replica. NFS 10GbE (1.1 GB/s, low cost) → Small scale, slow load tolerant. Distributed Block 25GbE (2-3 GB/s, high IOPS, high cost) → Medium-large. Object Storage + Local Cache (unlimited capacity, cache-dependent) → Recommended multi-tier: S3/MinIO as source of truth, node-local NVMe as cache, pull on startup.

9. Step 5: Network Assessment

9.1 North-South

~6 KB/request. Peak 12 KB/s. Focus on connection limits (Nginx worker_connections), SSE buffering ( proxy_buffering off), timeouts (300s).

9.2 East-West (Critical)

TP=4 all-reduce: 2 × (N-1)/N × Hidden × Dtype × Batch_Tokens. Prefill (8192 tokens) = 30 GiB. NVLink 3.0 → 60 ms; PCIe 4.0 → 1.2 s. Decode (16 concurrent) = 900 MiB/s. Mixed chunked prefill ~75 GB/s sustained → NVLink/RDMA mandatory.

9.3 Multi-Node

Pipeline Parallel (PP) cross-node: small activation transfers (134 MB) → 100G RoCE fine. Tensor Parallel cross-node: same 30 GiB prefill → needs 200G RoCE (1.4 s) or IB HDR/NDR. Architecture preference: TP intra-node (NVLink), PP/DP inter-node.

9.4 PD Disaggregation

KV transfer: 8192 tokens × 0.3125 MiB = 2.5 GiB/request. 2 req/s = 5 GiB/s sustained → Requires RDMA (200G RoCE 20% utilization). TCP 25GbE (2.5 GB/s) insufficient.

9.5 Storage Network

Isolate or QoS. tc example to limit NFS traffic. Risks: misconfiguration can break connectivity. Prefer switch-side QoS.

10. Practical Commands & Output Interpretation

10.1 Hardware Inventory

nvidia-smi --query-gpu=index,name,memory.total,...

Check memory.total (81920 MiB = full 80G), MIG partitioning, idle power (~60-80W A100). nvidia-smi topo -m for NVLink topology (NV12 = full mesh). nccl-tests/all_reduce_perf for busbw (A100 should hit 450-500 GB/s).

10.2 Model Load Benchmark

Clear page cache ( sync && echo 3 > /proc/sys/vm/drop_caches), then time cat *.safetensors > /dev/null. Local NVMe ~4.5 GiB/s (32s). >120s indicates bottleneck. Per-file breakdown to identify slow disks. Service log: Model loading took X GiB memory and Y seconds.

10.3 KV Cache Validation

vLLM logs: Available KV cache memory: 28.56 GiB, GPU KV cache size: 93,712 tokens, Maximum concurrency for 8,192 tokens per request: 11.44x. Convert to business context: 93,712 / 4000 = 23.4 concurrent @ P95. Anomalies: 0.xx GiB → quantize or add GPUs; 1.xx concurrency → same; OOM at startup → increase TP or quantize; high gpu_memory_utilization causes fragmentation OOM → keep 0.85-0.92. FP8 KV ( --kv-cache-dtype fp8) doubles capacity but may hurt long-context quality.

10.4 Runtime GPU Monitoring

nvidia-smi dmon -s pucvmet -d 2

. Key: sm (SM utilization), mem (memory controller utilization – decode should be 80-100%), pwr, mclk. Prometheus DCGM exporter metrics table with alert thresholds (VRAM >92% 5m warning, >96% 1m critical; Temp >80°C; XID errors immediate; Util <5% 15m).

10.5 Inference Service Metrics

vLLM exposes: Queue ( num_requests_waiting, num_requests_swapped >0 = VRAM pressure), Latency (TTFT, TPOT, queue, prefill, decode histograms), Throughput (prompt/gen tokens, success), Capacity ( gpu_cache_usage_perc, cpu_cache_usage_perc, gpu_prefix_cache_hit_rate). Top three: gpu_cache_usage_perc >0.90 → saturating; num_requests_waiting growing → throughput insufficient; TTFT P95 breakdown: queue time high → add replicas; prefill time high → check prompt length distribution or east-west bandwidth.

11. End-to-End Evaluation Workflow

Eight phases: 0) Pre-check (OS, kernel, driver, CUDA, GPU list, RAM, CPU NUMA, disk, NIC, Python, Docker), 1) Requirements Alignment, 2) Model Analysis, 3) VRAM Calculation, 4) Storage Planning, 5) Network Planning, 6) Candidate Solutions Comparison (3 options with pros/cons/cost/risk), 7) Test Environment Validation (benchmark checklist), 8) Production Rollout (canary, monitoring, rollback). Pre-check script and pass/fail criteria table provided. Solution comparison template: Conservative (8×A100 BF16 TP=4 DP=2), Balanced (4×A100 INT4 TP=4), Cost (4×L40S INT4 TP=4). Validation: vllm bench serve with realistic load (not --request-rate inf for production sizing).

12. Common Troubleshooting Paths

12.1 Startup Failures

CUDA OOM: Check weight load log, TP matches GPU count, gpu_memory_utilization headroom. Kill stray processes carefully ( nvidia-smi PID → ps → owner confirmation → kill -TERM → wait → kill -KILL → nvidia-smi --gpu-reset as last resort).

Slow Model Load: iostat -x 1 shows storage saturation. Fix: local NVMe cache, serial rolling updates, check RAID health.

Port Conflict: ss -lntp, lsof -i :port.

12.2 Runtime Performance Degradation

TTFT Creep: Check queue time vs prefill time. Queue high → scale out. Prefill high → check prompt length spikes or TP communication. Cache near 1.0 → restart or limit prefix cache.

Throughput Halved: Check mclk (memory clock) and gtemp. Drop from 1593 MHz → thermal throttling. Check XID errors. Root cause often cooling/PSU, not software.

12.3 VRAM Anomalies

OOM Despite Free VRAM: Fragmentation or unexpected growth. Causes: activation spike from long prompt, PyTorch allocator fragmentation, unbounded prefix cache. Fix: limit max tokens, restart, set prefix cache limit.

High gpu_memory_utilization Causes OOM: Leaves <5% for temporary allocations. Stay at 0.85-0.90.

12.4 Network Issues

Cross-Node TP Slow: Verify RDMA ( ibstat, nccl-tests multi-node). Busbw <5 GB/s → fell back to TCP. Fix: nvidia-peermem, PFC/ECN on switches, MTU 9000.

Chunked Streaming Output: Nginx proxy_buffering off missing. Config snippet provided.

13. Risk Assessment & Rollback Design

13.1 High-Risk Operations

Six operations with impact, pre-checks, risks, alternatives, backup, rollback, canary, verification: Model version change (keep old files, gray by user ID), gpu_memory_utilization tweak (test in staging), FP8 KV enable (eval on benchmark), Network QoS (test on non-prod NIC, switch-side preferred), Kill GPU processes (owner confirmation), Delete model files/images (move to trash first, verify backup).

13.2 Rollback Plan Template

Five elements: Trigger condition (e.g., TTFT P95 >5s 10m), Decision authority, Step-by-step commands (e.g., kubectl set image), Time estimate (must be measured), Post-rollback actions.

13.3 Change Management

Bastion-host access, approval tickets, config management (ConfigMap/Secret), low-traffic windows, 2× time buffer, canary (1 replica → 30 min observe), dual technical+business validation.

14. Production Operational Details

14.1 Config Management

Use Kubernetes ConfigMap for all runtime params (model path, TP size, max_model_len, GPU utilization, max_num_seqs, max_num_batched_tokens). Deployment YAML example with envFrom, resource limits matching TP, readiness/liveness probes with delays > model load time, hostPath for model cache (note: binds pod to node; prefer CSI local volumes).

14.2 Secrets Management

Never hardcode API keys. Use K8s Secret from file ( chmod 600), reference via valueFrom.secretKeyRef. Enable etcd encryption or external KMS. Minimize token scope (read-only for model pull).

14.3 Logging & Audit

Per-request: timestamp, user, model, prompt/gen tokens, TTFT, latency, finish reason, prefix cache hit. Per-change: who, what, old/new values. Per-exception: XID, OOM, kill, health fail. Logrotate with copytruncate (accepts minor loss) or create + SIGHUP. Separate audit logs (6mo-3yr retention) from debug logs (7-30 days).

14.4 Capacity Watermarks

Green: gpu_cache_usage_perc <0.60, queue=0. Yellow: 0.60-0.85 or occasional queue → plan expansion. Red: >0.90 or sustained queue or TTFT over budget → immediate action (throttle, scale, degrade). Scale-out (add replicas) preferred over scale-up (add GPUs) for marginal cost and redundancy. Validate max_num_seqs increases with benchmark.

14.5 Operational Hygiene

Daily GPU health check script (read-only): GPU count match, ECC errors, XID last 24h, temp/power/clocks, NVLink inactive links, VRAM waterline, stray processes. Script robustness: set -uo pipefail, quoted paths, independent checks, timestamps. Cross-verify monitoring vs patrol discrepancies.

15. Summary

15.1 Core Conclusions

VRAM balance: Limit = Weights + KV + Activations/Framework + Margin. Three must-know numbers: actual weight size ( du), per-token KV bytes (formula), framework overhead (1.5-2.5 GiB/GPU, non-sharding). GPU choice: bandwidth determines decode speed ( BW/Weight_Bytes), interconnect determines TP scalability (prefill comms = tens of GiB), precision support (FP8 needs SM89+). Storage bottleneck is load bandwidth, not capacity. Network bottleneck is east-west, not north-south.

15.2 Minimal Viable Assessment

du -sb

model dir → weight size

Read config.json: layers, KV heads, head_dim, hidden_size

Hand-calc per-token KV × Target_Concurrency × P95_Context → KV Need

Weight/TP + KV/TP + 2 GiB → Per-GPU Need → Round up to GPU SKU

Start service, read Available KV cache memory → Calibrate

Step 5 is truth; all hand math is estimation.

15.3 Top Five Mistakes

Only weight, forget KV → buy GPUs that serve 1-2 users.

Use max_model_len for concurrency → over-provision 2×.

Ignore non-sharding framework overhead → underestimate 16 GiB on 8 GPUs.

Assume PCIe ≈ NVLink → prefill comm 1.2s kills TTFT.

Ignore storage bandwidth → 22 min load breaks HA during rollout/recovery.

15.4 Future Shifts

Quantization advances (INT3/2), KV compression (MLA, sparse attention), PD disaggregation maturity (needs RDMA), Long-context growth (128K = 16× KV vs 8K). Methodology stable: Weight → KV → Comm → Calibrate.

15.5 Deliverables Checklist

Requirements sign-off, Model param sheet, VRAM calc sheet, 3-option GPU comparison, Storage plan, Network plan, Benchmark report, Deployment runbook, Rollback plan, Monitoring/Alert config (real exporter metric names), Patrol script. Each missing item becomes a future incident.