UniCBE: A Unified Multi‑Objective Optimization Framework for Contrastive Based Evaluation

The recent paper "UniCBE: An Uniformity‑driven Comparing Based Evaluation Framework with Unified Multi‑Objective Optimization" introduces a major breakthrough for Contrastive Based Evaluation (CBE) of large language models. The authors identify three fundamental challenges of existing CBE methods: sampling bias, unbalanced uncertainty reduction, and unfocused resource allocation in dynamic scenarios.

To address these issues, UniCBE constructs three decoupled sampling probability matrices—bias‑suppression, convergence‑acceleration, and expansion‑enhancement—each targeting one of the core objectives. The matrices are merged via a greedy sampling strategy and a Hadamard product to produce a unified sampling distribution that simultaneously optimizes accuracy, convergence speed, and scalability.

Extensive experiments on the AlpacaEval benchmark demonstrate that UniCBE achieves a Pearson correlation above 0.995 with only 83% of the annotation budget required by full evaluation. In dynamic settings where new models continuously join the pool, UniCBE reduces evaluation cost by more than 50% compared to baseline methods.

The paper also provides theoretical analysis showing that uniform allocation of preference budgets across model‑sample triples minimizes sampling bias and estimation error. Detailed ablation studies confirm the effectiveness of each matrix and the combined strategy. Generalization tests across different evaluators (GPT‑4‑turbo, GPT‑3.5‑turbo, Qwen‑Plus) and varying numbers of models and samples further validate the robustness of UniCBE.

Overall, UniCBE offers a systematic solution that unifies multi‑objective optimization for CBE, delivering higher evaluation accuracy, faster convergence, and superior scalability while substantially lowering annotation costs.