How LalaEval Revolutionizes Domain‑Specific LLM Evaluation

Pain Points

How can enterprises build domain‑specific large models?

How to flexibly generate evaluation sets to fully leverage internal data?

How to reduce subjectivity in human evaluation?

How to automatically detect low‑quality QA pairs and scores?

Solution

We propose LalaEval, a human‑evaluation framework that uses controversy analysis and score‑fluctuation analysis to automatically correct subjective errors, dynamically generate high‑quality QA pairs based on business scenarios, and guide domain LLM construction and iterative optimization.

Key Features

End‑to‑end domain LLM evaluation system, filling the evaluation gap in the freight domain.

Defines critical steps for framework design, question‑bank construction, scoring, and result output, ensuring high extensibility across domains.

Applies single‑blind testing to guarantee objective, fair scoring.

Introduces three analysis modules—evaluator controversy, item controversy, and score volatility—to automatically quality‑check scores, re‑identify low‑quality QA pairs, and quantify reasons for score fluctuations.

Framework Overview

LalaEval consists of five parts: domain scope definition, capability metric construction, evaluation set generation, evaluation standard formulation, and result statistical analysis.

1. Domain Scope Definition

Based on the MECE principle, we use a backward‑induction method to start from fine‑grained sub‑domains (e.g., intracity freight transportation) and iteratively aggregate to broader domains. Priorities are set (e.g., “intracity freight” as P0) to allocate evaluation‑set resources.

2. Capability Metric Construction

We build both generic and domain‑specific capability indicators. Generic indicators include semantic understanding, contextual dialogue, answer completeness and coherence, factual accuracy, creativity, and logical reasoning. Domain indicators cover concept and terminology understanding, company information, legal and policy knowledge, industry insights, company‑specific knowledge, and logistics‑environment creativity.

3. Evaluation Set Generation

We collect traceable source material, generate QA pairs with defined difficulty levels and capability dimensions, ensure each QA includes a question, a reference answer, and a source, and perform quality inspection before storage.

4. Evaluation Standard Formulation

Scoring uses a 0–3 scale where 0 indicates erroneous information and 1–3 assess correctness, completeness, and creativity. Single‑blind testing randomizes model responses for fairness. Evaluators are selected from domain experts and trained via example‑based methods, with pilot tests to resolve ambiguities.

5. Result Statistical Analysis

Scores are aggregated and normalized. Controversy analysis identifies evaluators and items with high disagreement, and score‑volatility analysis decomposes changes into four causes: item changes, model answer changes, evaluator inconsistency, and evaluator turnover.

Experimental Results

We evaluated OpenAI GPT‑4 (no web access), Baidu Wenxin Yiyan (with web access), and three fine‑tuned variants of ChatGLM2‑6B (different combinations of web access and RAG). Reported metrics include non‑zero answer ratios per capability, normalized average scores, score distributions, and disagreement degrees across capabilities, highlighting that creativity shows the highest disagreement.

Conclusion

LalaEval provides a systematic, automated, and extensible framework for domain‑specific LLM evaluation, addressing key pain points of model building, evaluation‑set generation, subjectivity reduction, and quality control.