How to Build High‑Quality Evaluation Datasets for AI Applications

Dataset Demand Analysis

The principle is to design a dataset that matches the evaluation goal: first define what to measure, then decide which data are needed.

Capability assessment : use comprehensive benchmarks such as MMLU to gauge knowledge breadth.

Problem定位 : create 100 multi‑step reasoning cases to locate reasoning bottlenecks.

Regression verification : maintain a golden dataset of 50 core scenarios for every iteration.

上线审批 : include 100 authentic customer‑service dialogues covering high‑risk cases.

竞品对标 : run the same dataset on both own and competitor models for direct comparison.

Dataset Construction Methods

Three data sources are combined according to evaluation goals.

Real‑data collection : logs, user feedback, historical tickets; high ecological validity but costly and privacy‑restricted.

Synthetic data generation : template filling, LLM‑generated variants, mutators for robustness testing; fills missing edge cases.

Public data selection : readily available benchmarks; useful for cross‑model comparison but may be contaminated or mismatched.

Real‑data collection & annotation

Source: online logs, user feedback, historical tickets (ensure compliance and de‑identification).

Sampling: stratified by scenario to cover high‑frequency, medium‑frequency, and long‑tail cases; avoid only "easy" or "hard" samples.

Annotation: label expected output, key entities, difficulty level; enforce clear guidelines and consistency checks.

Annotation consistency example : three annotators label the same dialogue with three different intent tags, indicating unclear guidelines and prompting a revision.

Synthetic data generation

Template filling : generate structured question variants (e.g., "Help me check order XXX").

LLM generation + human verification : let LLM produce diverse questions, then manually filter and correct.

Mutator : apply synonym substitution or perturbations to real data for robustness testing.

Combination generation : merge atomic scenarios into multi‑turn dialogues with compound intents.

Key principle: synthetic data must be manually verified before entering the evaluation set.

Public data filtering & adaptation

Filter public datasets for business‑relevant subsets.

Adapt formats to the evaluation schema.

Check for contamination (whether the model has seen the data during training).

Dataset Quality Assurance

Multi‑round QA process

Self‑check (dataset builder): ensure all required fields are filled.

Cross‑check (another annotator, random 20% sample): require consistency ≥90%.

Expert review (domain expert, high‑difficulty/high‑risk samples): no factual errors.

Trial run (evaluation engineer, known models): results must match expected distribution.

Bias detection & correction

Content bias : over‑representation of certain problem types; detect via scenario distribution statistics and supplement missing scenes.

Language‑style bias : overly formal or colloquial phrasing; analyze style diversity and add variants.

Label bias : annotator preference; compute inter‑annotator agreement (e.g., Kappa) and calibrate guidelines.

Temporal bias : dataset reflects outdated behavior; compare with recent logs and periodically inject fresh samples.

Example: 80% of questions are "query order" while real traffic shows only 40% order queries and 40% return‑exchange queries, indicating content bias that must be corrected.

Dataset Version Management & Continuous Update

Datasets require versioning to ensure reproducibility, traceability, and progressive evolution.

Semantic versioning : <major>.<minor>.<patch> (e.g., v1.2.0).

Snapshot storage : immutable files stored in S3/OSS with fixed paths.

Changelog : record every change (additions, removals, fixes). Example entry:

## v1.2.0 (2026-08-15)
- 新增：投诉场景样本15条（覆盖3种投诉类型）
- 优化：10条订单查询样本的问法多样性
- 修正：修正3条标注错误（修正前为“退款咨询”，修正后为“退货咨询”）
## v1.1.0 (2026-07-20)
- 新增：边界样本10条（含2条多轮对话）
- 移除：5条过时样本（对应已下线业务）

Link to model version : evaluation reports must state the dataset version used.

Continuous‑update strategy

New Badcase accumulation (≥50 samples) → batch annotation, weekly or bi‑weekly update.

Business changes → add or remove scenario samples as needed.

Data contamination detection → flag and replace immediately.

Periodic review → quarterly audit of dataset relevance.

The "data flywheel" shows how Badcase collection feeds systematic dataset updates.

Hands‑on Exercise: Rapid Dataset Design for a Specified Scenario

Students spend 15 minutes to design a complete dataset pipeline for a chosen scenario (e.g., intelligent Q&A客服).

Task assignment – instructor provides scenario description, evaluation goals, and template.

Group design – discuss and fill the framework (requirements, data sources, annotation schema, quality controls, version plan).

Quick showcase – each group presents core ideas for peer feedback.

The exercise template includes fields for demand analysis, data‑source ratios, annotation fields, consistency checks, and versioning schedule, with concrete numbers (e.g., 40% order queries, 30% returns, 20% complaints, 10% long‑tail).