What Three Days of Reproducing an ACL 2026 Paper Revealed About SFT Failures

Why I Spent Three Days Reproducing This Paper

As an engineer who regularly fine‑tunes language models, I noticed persistent "bad cases" where the model answered incorrectly despite similar training examples. The ACL 2026 paper by Tencent Mixed‑X and UNSW introduced the term Incomplete Learning Phenomenon (ILP), defining it as the failure of a fine‑tuned model to reproduce a portion of its own training data after convergence.

Across ten standard SFT datasets the authors measured an average ILP rate of ≈15.3% ± 2.1% . This is not a generalization issue, over‑fitting, or data noise; it is a blind spot hidden from traditional loss‑based evaluation.

My Minimal Reproduction with Qwen2‑7B + MedQA

I fine‑tuned Qwen2‑7B on the MedQA benchmark, achieving 72.3% eval accuracy, which seemed satisfactory. Applying the paper’s pass@5 detection to every training sample revealed that 18.4% of samples failed (none of the five generations were correct). These failures were uniformly distributed, not concentrated in high‑loss “hard” examples.

Manual inspection of 20 failed samples showed a "half‑understood" pattern: the model produced partially correct answers (e.g., correct drug name but wrong dosage, or correct diagnosis direction but missing key differentials). Single‑token cross‑entropy loss remained low, illustrating why loss alone cannot capture semantic‑level gaps.

Understanding MC Conversion

The core technique is Multiple‑Choice Conversion (MC conversion), which turns free‑text SFT evaluation into a multiple‑choice task. The workflow keeps the correct answer as option A, generates three plausible distractors (B/C/D) with GPT‑4, then samples each example five times (temperature 0.7, top‑p 0.9). If the correct answer appears in ≤1 of the five attempts (pass@5 < 0.2), the sample is marked as unlearned.

I tried three distractor‑generation strategies; their detection rates differed by about 8 percentage points, demonstrating that ILP detection reliability hinges on the quality of distractors . The paper’s hierarchical strategy—entity replacement, causal inversion, cross‑event confusion—provides a systematic, not ad‑hoc, approach.

Stability tests showed a Kappa agreement of 0.91 when all distractors were regenerated, ±0.5% variance across temperatures (0.5/0.7/0.9) , and ≈0.8% variance across models (Qwen vs. LLaMA) , confirming that ILP is a genuine phenomenon rather than a methodological artifact.

Ablation and Threshold Sensitivity

When I trained the model directly in MC format, the unlearned rate dropped dramatically to 4.2% , indicating that MC‑style training measures whether the model can answer the choice, not whether it has internalized the underlying knowledge.

Raising the pass@5 threshold from 0.2 to 0.4 reduced the reported unlearned rate to 9.7%, but many of those newly‑labeled “learned” samples remained unstable in production. This justifies the paper’s strict pass@5 < 0.2 criterion.

Five Root Causes – The Attribution Framework

The authors decompose ILP into five root causes using a 2×2 matrix: horizontal axis = whether the base model already knows the fact (zero‑shot probing), vertical axis = whether the SFT label is correct. Applying this to my MedQA failures, I found:

Root I : 8.3% of the total unlearned rate

Root II : 4.1% of the total unlearned rate

Together they account for 12.4%, over two‑thirds of all failures, revealing that simply adding more epochs does not address the underlying knowledge gaps.

Attribution is computed concretely: zero‑shot probing uses pass@5 on the base model; SFT label correctness is verified by manual review plus an LLM judge. This yields reproducible signals for each root cause.

Cross‑model attribution tests (Qwen, LLaMA, OLMo2) showed 86% agreement on roots I/II and 78% on roots III–V, indicating robustness of the framework.

Five Remedial Strategies and Compute Budget

For each root cause the paper proposes a targeted fix. My experiments produced the following improvements:

Root I (knowledge gap) : Continual Pre‑Training (CPT) on 5 B medical tokens raised performance by 11.8%, whereas extending training to 30 epochs yielded only 2.0%.

Root II (calibration) : CPT‑based calibration added 2.6%.

Root III (dynamic bucketing) : Better than outright deleting contradictory samples, preserving data volume.

Root IV (left‑side forgetting) : Global shuffling + resampling boosted ROUGE‑L by 27% for the top 10% of data, while extra epochs had negligible effect.

Root V (progressive epochs) : Incremental epochs contributed a modest 1.8% gain at near‑zero cost.

The compute cost for Root I’s CPT was roughly 5 B tokens on a single A100 GPU for ~48 hours, whereas the same compute spent on extra epochs produced <2% gain—an order‑of‑magnitude difference in cost‑effectiveness.

Re‑Evaluating Six Months of Past SFT Experiments

Using the ILP detection pipeline, I re‑examined all SFT models deployed over the past half‑year. A finance‑focused project exhibited many bad cases on recent financial events; the analysis traced them to Root I (out‑of‑date knowledge), suggesting CPT with fresh financial news as the remedy.

A dialogue‑bot project suffered from bad cases on early‑training topics; the attribution pointed to Root IV (left‑side forgetting). Applying global data shuffling and resampling resolved the issue without additional curriculum learning.

Why Supervised Fine-Tuning Fails to Learn: A Systematic Study of Incomplete Learning in Large Language Models
https://arxiv.org/pdf/2604.10079