Choosing the Right LLM: A Complete Guide to Selecting from Over 2 Million Models

Functional vs non‑functional dimensions

Functional requirement describes what a model can do (task coverage). Non‑functional requirement describes how well it does it (latency, cost, stability).

Core metric definitions

Performance and latency metrics

TTFT (Time to First Token) – time from request submission to receipt of the first token. It includes request queue time, pre‑fill time and network latency. Longer prompts increase TTFT because the pre‑fill stage must compute KV cache for the whole input. Typical thresholds: code completion needs TTFT < 100 ms; chatbot TTFT < 500 ms is acceptable.

Throughput (TPS / tokens per second) – tokens generated per second during the decode stage. This stage is memory‑bandwidth bound: the bottleneck is reading model weights from GPU HBM, not raw FLOPs.

ITL (Inter‑Token Latency) / TPOT (Time Per Output Token) – average interval between consecutive tokens. High ITL makes streamed output feel choppy even if TTFT is low.

E2E Latency (end‑to‑end) – total time from request receipt to full response delivery, roughly TTFT + total generation time. In agent pipelines it also includes retrieval, tool calls and post‑processing.

Goodput – effective throughput measured under a defined SLO; it reflects successful requests per second rather than raw token speed.

SLO (Service Level Objective) – target performance for a metric, e.g., 95 % of chat interactions must have TTFT < 200 ms.

Model capability terms

Benchmark – standardized tasks or datasets used to evaluate a model (e.g., MMLU, GPQA, SWE‑bench Verified, NoLiMa, HELMET, GRIT, CLIPScore, Chatbot Arena).

Reasoning model – trained with reinforcement learning to decompose problems into intermediate steps; typically slower TTFT and higher token count.

Multimodal – accepts non‑text inputs (audio, images, video) and aligns them with language output. High‑resolution images consume GPU memory comparable to thousands of tokens.

Context window – maximum token length a model can attend to in a single pass. State‑of‑the‑art models support up to 1 M tokens (≈75 k words), enabling whole‑document or full‑trace processing.

Token – basic text unit; Chinese ≈1‑2 characters per token, English ≈0.75 words per token. Token count drives API cost and latency.

KV Cache – caches key‑value matrices for shared prompt prefixes, avoiding recomputation. Claude reports up to 90 % fee reduction for cached prefixes.

Function calling / Tool use – model emits structured parameters (usually JSON) to invoke external functions or APIs; core capability for agents.

Tool Call Error Rate – proportion of tool calls that fail due to format errors, wrong tool selection, or hallucinated parameter values.

Structured Output – model output that conforms to a schema (JSON Schema, XML). Structured Output Error Rate measures violations of that schema.

MoE (Mixture of Experts) – activates only a subset of expert sub‑networks per inference, keeping overall parameter count high while reducing compute. DeepSeek V3 is an example.

RAG (Retrieval‑Augmented Generation) – retrieves relevant documents from a vector store and feeds them together with the query to the model, improving factuality and reducing hallucinations.

Hallucination – generated content that is plausible but false or fabricated. Evaluated with TruthfulQA, FactScore, etc.

Prompt Caching – service‑side KV‑cache exposure to users; repeated long prefixes receive caching discounts.

Functional requirements – matching tasks to capabilities

Task types and benchmark alignment

General knowledge & reasoning – evaluate on MMLU, GPQA; suitable for QA, summarisation, content generation.

Code generation & engineering – evaluate on SWE‑bench Verified (real GitHub issues); measures ability to fix bugs or write code.

Long‑document understanding – evaluate on NoLiMa, HELMET (128 K+ token contexts); measures extraction and reasoning over very long texts.

Multimodal understanding – evaluate with GRIT (visual QA) and CLIPScore (image‑text similarity); tests alignment across modalities.

Dialogue & instruction following – evaluate with Chatbot Arena (human head‑to‑head voting); reflects real user preference.

Tool‑call reliability

In agent workflows the observed Tool Call Error Rate can reach 21.52 % (≈1 failure per 5 calls). When calls are chained (e.g., 5‑step pipelines) the overall task success drops below 30 %.

Typical error sources:

Parameter format error – generated JSON does not match the schema.

Wrong tool selection – a semantically similar but functionally different tool is invoked.

Parameter value hallucination – format is correct but values are fabricated.

Reasoning models that can self‑reflect on failed tool results can retry, reducing hallucination and error rates.

Structured output impact

In downstream pipelines a Structured Output Error Rate of 5 % (1 error per 20 JSON outputs) forces additional error‑handling logic, especially in high‑risk domains such as medical diagnostics or financial compliance.

Non‑functional requirements – quality dimensions

Latency matching to scenarios

Real‑time dialogue, search autocomplete – TTFT is the primary metric.

Agent workflows where output is not shown directly – Throughput (TPS) is more important.

Voice interaction, translation – E2E latency dominates.

Reference thresholds: chat TTFT P95 < 0.5‑0.6 s (excellent), up to ~1 s acceptable; E2E latency P95 < 2‑3 s for ~200 token responses.

P50 reflects typical experience; P95/P99 capture tail latency that drives user complaints.

Context window sizing and cost trade‑offs

≤ 128 K tokens – fits most RAG, ordinary dialogue, medium‑length summarisation.

200 K‑500 K tokens – enables single‑call processing of full legal contracts, technical specifications, or medium codebases.

≥ 1 M tokens – supports long‑running agents that retain full tool‑call histories or legacy code modernisation.

Longer windows increase attention‑matrix complexity, GPU/TPU memory consumption and inference time.

Cost model (token‑based pricing)

Total cost of ownership (TCO) consists of:

API unit price (input + output tokens). Output tokens are typically 3‑5× more expensive than input tokens.

KV‑Cache hit rate – higher reuse of long, static prompts yields larger savings.

Retry cost – high tool‑call or structured‑output error rates can double effective spend.

Engineering integration cost – self‑hosted open‑source models avoid API fees but incur GPU procurement, ops, and quantisation effort.

Observability & compliance

Traditional APM metrics (latency, error rate) miss LLM‑specific failures such as hallucinations, retrieval of stale documents, or silent performance degradation on particular query types.

Production monitoring should additionally track:

Token‑level cost attribution (span‑level token counting).

Tool Call Error Rate.

Structured Output Error Rate.

Hallucination Rate (sample‑based factuality evaluation).

Data residency, SOC 2 Type II, HIPAA, GDPR compliance for regulated industries.

Systematic selection framework

Step 1 – Define hard constraints

Before comparing models, confirm non‑negotiable boundaries:

Does the context window accommodate the full input?

Is the cost within budget?

Can the latency meet user‑experience requirements?

Is integration with the existing stack feasible?

Only models satisfying all hard constraints enter the candidate pool.

Step 2 – Build a layered model matrix

Lightweight‑fast layer – high‑frequency, low‑complexity tasks (text classification, keyword extraction, simple QA, summarisation). Prioritise cost; TTFT < 0.5 s.

Flagship reasoning layer – complex, high‑value tasks (advanced code generation, multi‑step reasoning, deep document analysis). Prioritise quality; higher latency tolerated.

Open‑source self‑hosted layer – privacy‑sensitive or ultra‑cost‑optimised scenarios (e.g., Llama 4, Mistral, Qwen 3). Required for regulated data‑in‑place policies.

Step 3 – Validate with real production data

Benchmark scores alone are insufficient. Sample 100‑500 production requests covering typical, edge, and historically error‑prone cases. Compare candidate models on:

Accuracy (human review or LLM‑as‑Judge).

TTFT, E2E latency, Tool Call Error Rate, Structured Output Error Rate.

Actual token consumption per request.

Step 4 – Deploy routing and continuous monitoring

Successful deployments often route 2‑3 models intelligently: cheap model for simple tasks, balanced model for user‑facing dialogue, flagship model for critical decisions.

Recommended monitoring dashboard items (thresholds are illustrative):

TTFT P95 < 1 s – indicates queue or region issues.

E2E Latency P95 < 5 s – indicates orchestration or network problems.

Throughput > 50 tok/s – indicates hardware or concurrency limits.

Tool Error Rate < 5 % – agent reliability risk.

Output Error Rate < 2 % – downstream parsing risk.

Cost per 1 K requests growth < +20 % month‑over‑month – signals prompt bloat or abuse.

Hallucination Rate – set per scenario based on acceptable factual error.

Common pitfalls and anti‑patterns

Relying on benchmark rankings alone – benchmarks measure generic ability, not your specific prompts or data distribution. Always validate with real traffic.

Ignoring tool‑call error amplification – a 21 % single‑call error rate can cause >70 % failure in a 5‑step chain; prioritize reducing this error over raw model intelligence.

Treating latency as a single metric – average latency hides tail latency (P95/P99) that drives complaints. Both TTFT and throughput must be considered in context.

Underestimating KV‑Cache and prompt design impact on cost – identical models with different prompt reuse or token compression can differ by several‑fold in cost.

Assuming a one‑time selection is permanent – most LLM improvements come from tooling and inference extensions rather than core architecture changes. Re‑evaluate primary candidates quarterly.