How Huolala Built LaLaEval: A Practical Framework for Large Model Evaluation

01 Application Background

Large models have evolved from early expert systems to today's deep‑learning era, with a surge of models in 2023. Their practical value lies in solving real problems, leading to over 200 models in early 2024 across domains such as marketing, customer service, and data analysis. However, challenges remain, including domain‑specific knowledge depth, timeliness of knowledge, personalization, cost and speed, and stability and security.

These issues can be framed as three evaluation dimensions: benefit, cost, and risk. Systematic evaluation helps balance them.

02 Evaluation Framework (LaLaEval)

The framework follows the typical model‑to‑production pipeline: define scenario needs, design model and engineering solutions, select a base model (balancing capability, efficiency, and cost), implement AI agents with RAG, and conduct multi‑stage evaluation (offline, online A/B, and continuous monitoring).

Evaluation is divided into three phases: (1) base model selection, (2) offline effectiveness verification, and (3) online validation (A/B testing, not covered in detail). The first two phases constitute the LaLaEval framework.

LaLaEval consists of five steps:

Define the domain scope and boundaries.

Design detailed metrics to assess model capabilities.

Generate evaluation datasets aligned with the domain.

Establish rigorous evaluation standards.

Statistically analyze results, checking for scorer and item disputes, and perform second‑level quality checks.

Domain boundary definition relies on expert knowledge and follows the MECE principle to ensure exhaustive, non‑overlapping sub‑domains. Once boundaries are set, capability indicators (general abilities like semantic understanding and reasoning, plus domain‑specific knowledge such as logistics policies) are identified.

Dataset creation follows a standard pipeline: collect raw corpora, design QA pairs, generate dialogue sets, and perform quality checks before ingestion.

Scoring standards are quantified to enable blind, multi‑blind testing, reducing subjectivity. Dispute analysis identifies items with high scorer variance for re‑review.

03 Application Practice

Two case studies illustrate LaLaEval in action.

Case 1: Freight‑Domain Large Model

The goal is to build a model that answers logistics‑specific queries, understands industry terminology, corporate information, legal policies, and provides strategic insights. Evaluation metrics cover factual accuracy, creativity, and business impact, with results aggregated across dimensions.

Iterative testing showed factual accuracy reaching 93% for the third model, surpassing GPT‑4.

Case 2: Invitation (Driver Onboarding) Model

The invitation process involves driver registration, credential verification, and membership conversion. An AI invitation agent uses ASR to transcribe driver speech, feeds it to the model, and TTS to respond, aiming to automate the workflow.

Key evaluation metrics include response latency, compliance (no red‑line content), factual correctness, and conversion‑driving dialogue quality.

Simulated driver interactions are designed with varied personas, scenarios, and question sets to mimic real‑world conditions.

After simulation, structured evaluation follows the same LaLaEval steps, employing blind scoring, dispute analysis, and quality checks.

04 Summary and Outlook

Current evaluation still relies heavily on manual effort—simulated dialogues, scorer involvement, and data handling. Future directions aim for greater automation through more extensive model participation, productized evaluation tools (e.g., “LaLa Smart Evaluation”), and end‑to‑end pipelines that reduce human bottlenecks.

Automation, platformization, and intelligent evaluation are expected to become the norm, making large‑model assessment more efficient and scalable.