Can an 8B Model Outperform GPT‑4 in Faithfulness Detection? Inside FaithLens

Background

Large language models (LLMs) excel at retrieval‑augmented generation (RAG) and summarization, but they often produce faithfulness hallucinations —outputs that contradict or are unsupported by the supplied reference documents. Such errors are critical in high‑risk domains (law, medicine, finance).

Problem

Existing detectors face a trade‑off: ultra‑large models (e.g., GPT‑4, o1) achieve high accuracy but are expensive and slow, while lightweight detectors (e.g., MiniCheck‑7B) are fast but only emit a binary label, offering no rationale for the decision.

Solution: FaithLens

FaithLens is an 8‑billion‑parameter hallucination‑detection model that jointly produces a detection label and a natural‑language explanation. It matches or exceeds the performance of top‑tier large models on twelve diverse benchmarks while keeping inference cost low.

Key Contributions

Performance breakthrough for small models: FaithLens outperforms GPT‑4.1, OpenAI o3 and other super‑large models across 12 tasks covering RAG, summarization, multi‑hop QA, and fact‑checking.

White‑box, explainable detection: The model emits a high‑quality textual explanation that pinpoints the source of the hallucination, improving user trust.

Reinforcement‑learning‑driven explanation optimization: A novel “explanation‑quality reward” measures whether the generated explanation enables a novice model to predict the correct label, encouraging clearer, evidence‑rich reasoning.

Method

1. Data synthesis and cleaning

Open‑source hallucination datasets usually contain only binary labels. FaithLens first uses a strong reasoning model (DeepSeek‑V3.2‑Think) to generate synthetic examples with chain‑of‑thought (CoT) reasoning and explicit explanations. To filter noisy synthetic data, a three‑dimensional strategy is applied:

Label correctness: Discard samples whose model‑predicted label disagrees with the ground‑truth.

Explanation quality: Compute perplexity (PPL) of a target model (e.g., Llama‑3.1‑8B‑Inst) with and without the explanation; a significant PPL reduction indicates a useful explanation.

Data diversity: Embed all samples, cluster them with K‑Medoids, and construct a “probe set” of core samples that improve perplexity for other samples in the same cluster, thereby enhancing cross‑task generalization.

2. Rule‑based reinforcement learning (Rule‑Based RL)

After supervised fine‑tuning on the filtered synthetic data, FaithLens undergoes a rule‑based RL stage using the GRPO algorithm. Three reward signals guide training:

Prediction correctness reward: +1 for a correct hallucination label, 0 otherwise.

Explanation‑quality reward: Feed the generated explanation to an un‑fine‑tuned “novice model” (Llama‑3.1‑8B‑Inst). If the novice model then predicts the correct label, reward +1; else 0. This forces explanations to be sufficiently informative for a beginner.

Format reward: Enforces the required output structure (label + explanation).

Experiments

Detection performance

FaithLens is evaluated on 12 cross‑domain benchmarks (news summarization, RAG QA, fixed‑document QA, fact‑checking, multi‑hop reasoning) drawn from LLM‑AggreFact and HoVer. It achieves the highest average scores, surpassing GPT‑4.1 and o3 despite using only 8 B parameters. Compared with specialized detectors such as MiniCheck and ClearCheck, FaithLens shows superior accuracy and the lowest performance variance across tasks, indicating strong robustness and generalization.

Explanation quality

Human judges and automatic GPT‑4.1 evaluations assess readability, helpfulness, and informativeness. FaithLens explanations are consistently clearer, more specific, and can pinpoint concrete hallucination causes (e.g., “fact not present in document”, “incorrect causal inference”, “numeric distortion”) rather than providing vague repetitions.

Inference cost

Because FaithLens is an 8 B model, its GPU memory footprint and latency are far lower than API‑based closed‑source giants, delivering substantially reduced inference cost while maintaining better performance.

Ablation and case studies

Systematic ablations that remove the three‑dimensional filter, the explanation‑quality reward, or the RL stage each cause noticeable drops in detection and explanation metrics, confirming their importance. Case studies on long documents and multi‑hop reasoning show FaithLens aligning evidence more precisely than GPT‑4o or o1, splitting reasoning into clear steps (e.g., confirming attribute presence, then locating contradictory dates).

Novice‑model selection

Using a novice model from the same family (Llama‑3.1‑8B‑Inst) yields the strongest reward signal. Cross‑family models (e.g., Qwen‑2.5‑7B‑Inst) introduce a language‑style gap that degrades reward accuracy, highlighting the benefit of a shared “language” between teacher and student.

Resources

Paper: https://arxiv.org/abs/2512.20182

Code repository: https://github.com/S1s-Z/FaithLens

Code example

收
藏
，
分
享
、
在
看
，
给
个
三
连
击呗！