Which LLM Is Best for Coding? Speed, Hallucination, and Context Compared

With major tech companies releasing a variety of large language models (LLMs), selecting the right one for your needs can be confusing. This article defines useful comparison metrics and explains how to leverage each model effectively.

Metrics for Comparing LLMs

Speed – How fast the model generates responses, measured in tokens per second (TPS).

Hallucination Rate – The frequency of incorrect or misleading answers; lower is better. Data sourced from the GitHub hallucination leaderboard.

Context Window Size – Amount of text the model can process at once; larger windows allow handling more complex projects.

Coding Performance – Ability to solve coding tasks, evaluated with benchmarks such as HumanEval+.

Key Benchmarks

HumanEval+ – Measures LLM ability to solve Python coding problems within limited attempts (score out of 100).

ChatBot Arena – Ranks LLMs based on real user feedback and voting.

Aider Multilingual Benchmark – Assesses multi‑language coding and debugging capabilities.

Which LLM Suits Your Coding Tasks?

GPT-4o – HumanEval+ 87.2, Speed 53.20 ± 15.57 TPS, Hallucination 1.5 %, Context 128K tokens.

GPT-4o mini – HumanEval+ 83.5, Speed 62.78 ± 19.72 TPS, Hallucination 1.7 %, Context 128K tokens.

o1 – HumanEval+ 89, Speed 134.96 ± 35.58 TPS, Hallucination 2.4 %, Context 10 K tokens.

o1‑mini – HumanEval+ 89, Speed 186.98 ± 47.55 TPS, Hallucination 1.4 %, Context 10 K tokens.

o3‑mini – Speed 155.01 ± 45.11 TPS, Hallucination 0.8 %, Context 10 K tokens.

Gemini 2.0 Flash – Speed 103.89 ± 23.60 TPS, Hallucination 0.7 %, Context 100 K tokens.

Gemini 1.5 Flash – HumanEval+ 75.6, Speed 112.57 ± 24.03 TPS, Hallucination 0.7 %, Context 100 K tokens.

Gemini 1.5 Pro – HumanEval+ 79.3, Speed 45.47 ± 7.78 TPS, Hallucination 0.8 %, Context 1‑2 M tokens.

Claude 3.7 Sonnet – Speed 46.43 ± 7.35 TPS, Hallucination – , Context 20 K tokens.

Claude 3.5 Sonnet – Speed 43.07 ± 7.03 TPS, Hallucination 4.6 %, Context 20 K tokens.

Claude 3.5 Haiku – Speed 42.90 ± 6.83 TPS, Hallucination 4.9 %, Context 20 K tokens.

Overall, no single model excels in every category. Based on our benchmarks, the leaders are:

Lowest hallucination rate: Gemini 2.0 Flash

Highest speed: GPT‑4o‑mini, Gemini 1.5 Flash, Gemini 2.0 Flash

General intelligence (non‑reasoning): GPT‑4o, Claude 3.5 Sonnet, Claude 3.5 Haiku, Gemini 1.5 Pro

Reasoning‑capable intelligence: Claude 3.7 Sonnet, o1, o1‑mini, o3‑mini

Local LLM Options

If you need an offline AI assistant or want to avoid sharing code with API providers, you can use local models via Ollama or LM Studio.

Currently the strongest models are Qwen‑2.5‑Coder and DeepSeek R1, but any sufficiently small model from the Ollama suite can be used depending on your hardware.

Author: 场长

Related reading:

A Brief History of LLMs: From Transformer (2017) to DeepSeek‑R1 (2025)

Mistral AI Releases Small, Powerful Open‑Source AI Models

Deploy Your Own DeepSeek Model Locally and Build an AI Application Platform