Demystifying DeepSeek‑V4 Benchmarks with Real‑World Data

Quick Reference of Evaluation Metrics (Bookmark)

The evaluation covers six ability dimensions and their representative benchmarks:

World Knowledge – MMLU, SimpleQA

Language Reasoning – BBH, DROP

Programming – LiveCodeBench, Codeforces, HumanEval

Mathematical Reasoning – GSM8K, MATH, IMOAnswerBench

Long Context – LongBench, MRCR, CorpusQA

Agent – SWE‑bench, Terminal Bench, Toolathlon

1. Programming Ability: V4 Takes First Place, How Measured?

LiveCodeBench 93.5 (vs Opus‑4.6 88.8) – real‑time coding competition using live LeetCode/Codeforces problems; a score of 93.5 means the model solves 93.5 of 100 problems.

Codeforces 3206 (vs GPT‑5.4 3168) – competitive‑programming ranking; 3206 corresponds to a "gold" human level.

What Do These Benchmarks Test?

LiveCodeBench

Measures model performance on live programming contests, preventing memorization of static test cases.

Codeforces

Measures the model's ranking among human competitors in algorithmic contests.

HumanEval

Evaluates code generation from function descriptions. Example:

示例：
描述："写一个函数，判断一个数是否为质数。"
模型输出：
def is_prime(n):
    if n <= 1:
        return False
    for i in range(2, int(n**0.5) + 1):
        if n % i == 0:
            return False
    return True

Score 76.8 means 76.8% of 164 problems passed.

2. Mathematical Reasoning: V4 Surpasses at Olympiad Level

IMOAnswerBench 89.8 (vs Opus‑4.6 75.3, +14 points) – International Math Olympiad‑level problems.

Apex Shortlist 90.2 (vs Gemini 89.1) – high‑school competition problems.

What Do These Benchmarks Test?

GSM8K – elementary school math word problems; e.g., "小明有 50 元，买了 3 支铅笔，每支 5 元，还剩多少钱？" → 35 元.

MATH – high‑school competition problems from AMC, AIME, etc.

IMOAnswerBench – extremely difficult problems only top math students can solve.

3. Knowledge: V4 Chinese Knowledge Ranks Second to Gemini

Chinese‑SimpleQA 84.4 (second to Gemini 85.9).

SimpleQA‑Verified 57.9 (vs Opus‑4.6 46.2).

What Do These Benchmarks Test?

MMLU – 57‑subject breadth; example question about photosynthesis.

SimpleQA – factual question answering; e.g., "珠穆朗玛峰有多高？"

4. Agent Ability: V4 Near Parity with Opus‑4.6

SWE‑bench Verified 80.6 (vs Opus‑4.6 80.8, gap 0.2).

What Do These Benchmarks Test?

SWE‑bench – solving real GitHub issues.

Terminal Bench – executing tasks in a command‑line terminal.

Toolathlon – coordinating multiple tools to complete complex tasks.

5. Long‑Context: V4 Ranks Second to Opus‑4.6

MRCR 1M 83.5 (vs Opus‑4.6 92.9) – "big‑sea‑needle" retrieval in 1 million‑token texts.

CorpusQA 1M 62.0 (vs Opus‑4.6 71.7) – question answering over a 1 million‑token document.

What Do These Benchmarks Test?

MRCR – find a specific piece of information in a massive text.

CorpusQA – read an entire book and answer a question about it.

6. Hidden Efficiency Boosts (3.7‑13.7×)

Beyond raw capability, V4 reduces compute and memory usage dramatically.

Token‑level compute : V4‑Pro is 3.7× lower than V3.2; V4‑Flash is 9.8× lower.

KV‑Cache memory : V4‑Pro is 9.5× smaller; V4‑Flash is 13.7× smaller.

Practical impact: faster inference, lower deployment cost, and lower latency.

7. Three Inference Modes: Choose to Cut Cost in Half

Non‑think – fast, cheap; for simple Q&A or chat.

Think High – balanced quality and speed; for code generation and document writing.

Think Max – maximum reasoning; for math proofs, extreme reasoning, and Agent tasks (SWE‑bench + tool use).

你的任务是什么？
│
├─ 简单问答 / 闲聊
│   └─ Non‑think（快速、便宜）
│
├─ 代码生成 / 文档写作
│   └─ Think High（平衡质量和速度）
│
├─ 数学证明 / 复杂推理
│   └─ Think Max（质量优先）
│
└─ Agent 任务（SWE‑bench 类）
    └─ Think Max + 工具调用

8. One‑Sentence Summary of DeepSeek‑V4 Evaluation

Programming – 🥇 First (LiveCodeBench 93.5, Codeforces 3206).

Mathematics – 🥇 First (IMOAnswerBench 89.8, Apex 90.2).

Chinese Knowledge – 🥈 Second (Chinese‑SimpleQA 84.4, just behind Gemini).

Agent – 🥈 Second (SWE‑bench 80.6, tied with Opus‑4.6).

Long‑Context – 🥈 Second (MRCR 1M 83.5).

General Knowledge – 🥉 Third (MMLU‑Pro 87.5, on par with GPT‑5.4).

Key takeaway: mastering the five core metrics—MMLU, HumanEval, GSM8K, LongBench, SWE‑bench—lets you quickly gauge a model's true strength.