Deep Guide to LLM Performance Testing and Optimization

Why Traditional Testing Misses LLM Performance Issues

As domestic LLMs such as ChatGLM, Qwen and DeepSeek move into finance, government and healthcare, 83% of deployed applications experience performance degradation—doubling latency, dropping throughput, and frequent GPU memory leaks (2024 China AI Engineering Whitepaper). The article argues that model inference is a dynamic compute‑graph and hardware‑resource competition, not a simple function call, and that prompt engineering affects GPU utilization.

Three Common Misconceptions in LLM Performance Testing

Misconception 1: Low latency equals good performance. A bank’s risk‑assessment assistant showed a P95 latency of 320 ms, but under >20 QPS the GPU OOM rate rose to 47%, causing a service avalanche. The root cause was testing only single‑request latency without a stepped load curve (5/10/20/50 QPS) and without monitoring GPU Memory Utilization or TensorRT cache hit rate.

Misconception 2: Standard datasets are the gold standard. Using Alpaca or ShareGPT to evaluate a medical Q&A model reported an average token generation speed of 128 tok/s, yet real‑world queries containing domain terms like “PD‑L1 expression‑positive rate” dropped to 31 tok/s because of long‑context recomputation. The article recommends building domain‑specific load‑pattern (DSLP) test sets.

Misconception 3: Quantization always speeds up models. A team quantized a 7 B model from FP16 to INT4, expecting a 2.3× throughput gain, but observed an 18% drop. The vLLM deployment lacked PagedAttention, causing INT4 weight loading to increase PCIe bandwidth contention. Optimization must consider the inference engine, hardware topology, and scheduling strategy.

Four‑Dimensional LATC Metric Framework

Latency : Separate first‑token latency (TTFT) from per‑token latency (TPOT). TTFT reflects prompt processing and KV‑cache initialization; TPOT shows decoder compute density.

Availability : Beyond HTTP 5xx, monitor CUDA context restarts and vLLM scheduler queue timeouts.

Throughput : Measure “effective requests per second” rather than raw token/s, excluding empty generations and duplicate stop tokens.

Cost : Track GPU‑hour consumption, peak memory usage, and network I/O; a client reduced monthly GPU cost by 22% after removing redundant system prompts that added 15% unnecessary requests.

Practical Optimization Case Study

Scenario: A government knowledge‑base LLM exceeded the latency target of <800 ms, averaging 1 420 ms.

Step 1 – Precise Attribution: Using NVIDIA Nsight Systems together with the vLLM Profiler, the bottleneck was identified in the RAG retrieval module, where vector similarity computation consumed 63% of time and ANN search (FAISS IVF_PQ) was not enabled.

Step 2 – Layered Optimization:

Inference layer: Enable FlashAttention‑2 and PagedAttention, reducing KV‑cache fragmentation and improving TPOT by 37%.

Retrieval layer: Switch Faiss index from Flat to IVF1024,PQ16, cutting TTFT by 210 ms.

Cache layer: Build an LRU‑2 cache for high‑frequency policy queries (e.g., “individual‑income‑tax special deduction standard”), achieving a 68% hit rate and stabilizing request latency at 210 ms.

Step 3 – Closed‑Loop Validation: Design a mixed load test (“spike + long‑steady”) that ramps QPS from 5 to 50 within 10 seconds and holds for 30 minutes. After optimization, no memory leaks occurred and GPU memory fluctuation stayed below 5%.

Engineering a Reproducible LLM Performance Testing Pipeline

The article proposes a three‑tier pipeline to replace ad‑hoc scripts:

Unit level: Use Litellm or OpenLLM to create a mock server that injects controllable latency or errors, validating client retry and fallback logic.

Integration level: Deploy Locust with a custom LLM task plugin that dynamically generates prompt templates, simulates session‑aware load, and automatically collects LATC metrics.

Production level: Connect Prometheus and Grafana to the vLLM Metrics API, setting alerts such as TTFT > 1 s or GPU memory > 90% to achieve minute‑level performance degradation detection.

Conclusion

Performance is not an afterthought for LLMs; it is a first‑principle requirement. The ultimate goal of LLM testing is to ensure models answer reliably, efficiently, and economically. When a single line of code can shift P99 latency by 300 ms or a prompt tweak can double GPU memory usage, engineers must treat GPU utilization like code coverage, token generation speed like API response time, and request cost like technical debt. Only then can large models evolve from flashy demos to trustworthy infrastructure.