Why AI Evaluation Is So Hard: Non‑Determinism, Human‑Like Challenges, and Benchmark Limits

7.1 Non‑Determinism

The core challenge of AI evaluation is non‑determinism: the same input can produce different yet reasonable outputs, breaking the traditional test assumption of deterministic input→output mapping.

7.1.1 Core dilemma – identical input, different output

Traditional testing expects a single correct answer, but AI systems are probabilistic, so multiple valid responses exist.

7.1.2 Measuring quality of probabilistic outputs

Multiple sampling evaluation : run the same input many times and assess the distribution of outputs.

Confidence intervals : report results with confidence ranges to reflect uncertainty.

Statistical significance testing : determine whether differences between models are statistically meaningful.

Key insight : non‑determinism is an AI "feature", not a bug, requiring redesigned evaluation methods.

7.2 Anthropomorphisation Challenge

AI is built to interact like a human, creating a paradox: we judge it by human standards while measuring it with machine metrics.

7.2.1 Subjective quality is hard to quantify

Human judgments of "good" are highly subjective; different judges (LLM‑as‑a‑Judge, humans) produce widely varying scores.

7.2.2 Context dependence

"You are too slow" is a complaint in a customer‑service chat but a tease among friends.

A humorous answer is a highlight in creative writing but unprofessional in legal advice.

Evaluation must consider context, yet the space of possible contexts is unbounded.

7.2.3 Long‑tail scenarios

AI must handle rare, out‑of‑distribution queries (e.g., "my order disappeared but I was charged"). Although they represent ~1% of traffic, errors can be severe.

Standard benchmark datasets cannot fully cover these cases; the evaluation focus shifts from "common‑question correctness" to "risk of catastrophic failure".

7.3 Data and Benchmark Limitations

7.3.1 Static benchmark vs. dynamic capability

Benchmarks become "dead" once published—fixed questions and answers—while AI models continuously evolve, leading to benchmark saturation, obsolescence, and over‑fitting to test sets.

7.3.2 Data pollution and over‑fitting risks

Pre‑training contamination : benchmark data appear in training corpora, inflating scores.

Instruction‑tuning contamination : benchmarks are reformatted as training samples, causing models to "memorise" test patterns.

RLHF contamination : reward models encode benchmark‑specific tricks, making models chase test‑set tricks rather than real ability.

These issues turn benchmarks from ability tests into memory tests.

7.4 Evaluation Cost and Efficiency Balance

7.4.1 Four cost dimensions

Compute cost : model inference and repeated sampling cost thousands to tens of thousands of dollars per full evaluation.

Human cost : annotators, expert reviewers, auditors cost tens of thousands to hundreds of thousands of dollars per project.

Time cost : end‑to‑end evaluation (data build → execution → analysis → iteration) takes weeks to months.

Opportunity cost : delayed releases due to evaluation bottlenecks can be substantial.

7.4.2 Efficiency vs. quality tension

Fast, lightweight evaluations (e.g., golden datasets) enable rapid iteration but have limited coverage; deep, full‑scale evaluations provide thoroughness but cannot keep pace with model updates.

Automation (LLM‑as‑a‑Judge, automated benchmarks) offers speed and repeatability but cannot assess subjective quality or contextual understanding, whereas manual evaluation is accurate but expensive and unscalable.

7.4.3 Pragmatic balance strategy

Run a "golden dataset" (100‑500 samples) on every iteration for quick sanity checks.

Execute automated full‑benchmark runs weekly or monthly, supplemented by LLM‑as‑a‑Judge for medium‑cost coverage.

Conduct expert‑driven deep evaluations quarterly or at major releases to assess subjective quality, safety, and fairness.

Key insight : the core constraint is not "can we measure?" but "is it worth measuring?"—evaluation design is a prioritisation problem under limited resources.

7.5 Practical Case: AI Customer‑Service Evaluation in E‑Commerce

A fictional e‑commerce team faces all the above difficulties:

Non‑determinism : the same user query receives divergent answers (one praised, one complained), making binary right/wrong judgments impossible. Solution: measure task completion rate instead of answer correctness.

Subjective quality dispute : product owners deem answers "too stiff", operators call them "professional". Solution: small‑scale human calibration to unify scoring, then scale with LLM‑as‑a‑Judge; fallback to "was the user problem resolved?" as a baseline.

Long‑tail coverage : 1% rare issues (e.g., missing order with charge) cause severe complaints. Solution: define high‑priority cases and a separate long‑tail testing tier, covering all with high‑priority and sampling long‑tail periodically.

Benchmark over‑fit : public benchmark‑selected high‑scoring model performs poorly in production. Solution: combine 20% public benchmark with 80% self‑built real‑world dataset.

Cost explosion : full evaluation consumes 3 GPU‑days and a week of analyst time. Solution: layered strategy—golden set per iteration → weekly automated run → quarterly deep manual review.

The case illustrates that AI evaluation has no perfect solution; the goal is to make the best trade‑offs within constraints.

7.6 Course Design Summary

Allocate 30 minutes across the sub‑modules, progressing from non‑determinism to anthropomorphisation, benchmark limits, and cost‑efficiency, ending with the practical case. Emphasise that each difficulty has actionable directions rather than pure complaints, and that the overall evaluation design is a value statement about what risks to prioritize.