Task‑Aware Decoding (TaD): A Plug‑and‑Play Method to Mitigate Hallucinations in Large Language Models

Large language models (LLMs) have ushered in a new era of AI capabilities, yet their tendency to generate inaccurate or fabricated content—known as hallucination—poses a major obstacle for reliable deployment. While Retrieval‑Augmented Generation (RAG) can alleviate hallucinations by injecting external knowledge, it does not fully address the intrinsic hallucination of the LLM itself.

To tackle this, JD Tech and Tsinghua University propose Task‑aware Decoding (TaD), a method that compares the output probability distributions of an LLM before and after supervised fine‑tuning. By constructing a knowledge vector from the differences, TaD amplifies factual knowledge learned during fine‑tuning while suppressing spurious patterns, and can be applied to any LLM without modifying its parameters.

The paper surveys existing hallucination sources—data quality, training procedures, and inference strategies—and reviews related mitigation approaches such as data cleaning, knowledge editing, and RAG. It highlights the limitations of these methods, especially their lack of generality and potential to degrade model performance.

Experimental results on multiple-choice, commonsense QA, and more challenging reasoning benchmarks show that TaD consistently outperforms baseline decoding strategies and other contrastive decoding methods. Gains are especially pronounced when fine‑tuning data are scarce, demonstrating TaD’s ability to bridge the gap between pre‑trained knowledge and downstream task requirements.

In a real‑world deployment, TaD is integrated with RAG in JD’s general‑purpose knowledge‑question answering system. The combined TaD+RAG pipeline achieves markedly lower hallucination rates while maintaining high answer relevance, confirming the practicality of the approach.

The authors conclude with a forward‑looking discussion, suggesting that future systems will likely combine RAG, autonomous agents, and advanced memory modules, and that deeper fusion of external knowledge with LLM reasoning—as exemplified by TaD—will remain a key research direction for achieving low‑hallucination AI.