15 Pitfalls in Training JEV Decision Models: From Pointer Heads to Data Leakage

Background

The author trained JEV decision models (0.8B, 2B, 4B) on Qwen3.5 for the Jev Decision Index. The model outputs probabilities for candidates in a single forward pass without autoregressive generation. Three task types: yes/no (noul), single-choice (choice), ordered scoring (score). A unified model later added a span head for information extraction.

1. Model Architecture

1.1 Category Count Limit

Initial training used 26 categories (A–Z logits at last position). This fails when candidates exceed 26 (e.g., BANKING77 with 77 classes, CLINC150 with 151).

1.2 Breaking the Limit

Three mainstream solutions from leaderboard analysis:

Group elimination (StartLux): split candidates into groups of ≤25, run model per group, keep top‑k, then final round. Drawbacks: multiple forward passes, correct answer may be eliminated early, eliminated items share a tiny probability (0.001) not from joint scoring.

Extended encoding + dedicated readout matrix : encode AA, AB… up to 255 codes, add a 255×hidden readout matrix initialized from token embeddings. Used by Perplexity Decider, Sieve, Fastino GLiDE.

Pointer head : place a special marker at each candidate end and a decision marker at question end; score via dot product of candidate‑marker and decision‑marker hidden states:

logit_k = <W_c·LN(h_k), W_q·LN(h_decide)> / √256 + <w, W_c·LN(h_k)>

. Candidate count limited only by context length.

Author chose pointer head. Candidates still written as A: key — description (AA, AB beyond 26) for human readability, but scoring ignores label tokens. CLINC150’s 151 classes now fit in one forward pass. Cost: new head randomly initialized, requires training (planned from base anyway).

Tip: record marker token positions during concatenation, not by searching text later; delete any user‑input strings matching marker tokens to avoid mis‑location.

1.3 Bidirectional Attention from Step 1

Pointer head defect: causal attention prevents candidate A from seeing B, C. Only decision marker sees all. Leaderboard models use bidirectional attention (StartLux: prompt‑level; Verdin: chunked state‑level). Qwen3.5 has hybrid layers: 1 full‑attention layer per 4 layers, others linear attention (Gated DeltaNet). Solution: treat layer types separately:

Full‑attention layers: bidirectional between context and final decision marker.

Linear‑attention layers: keep causal.

Implemented by passing a per‑layer‑type mask dictionary to Qwen3_5TextModel without modifying transformers source.

Two critical lessons:

Bidirectional must be on from training step 1. Opening mask at inference on a causally‑trained model does not improve results. Author re‑trained from base with final structure.

Unit‑test mask effectiveness. In bidirectional mode, earlier token hidden states should change when later candidate content changes; in causal mode they should not. Silent bugs only degrade metrics.

Bonus: "multiple questions per sequence" — state written once, then each question’s instruction, candidates, decision marker; one forward pass answers all, multi‑question cost ≈ single question.

Result: pointer‑head 4B re‑trained from base raised public Jev Decision Index from 41.12 to 53.40 (rank 108/249).

1.4 Shuffling Option Order

First 4B showed strong position bias: on binary choice, model favored second option (BPoMP 85%, Humicroedit 78% vs true 50/50). Two fixes:

Inference‑time debias : run forward and reverse order, average probabilities. No retraining, 2× compute. On 2,500 samples: avg acc +1.6, BPoMP +19, Humicroedit +12; public index 39.41 → 41.12.

Training‑time shuffle : randomize candidate order for choice and noul (score is ordered, cannot shuffle). Root‑cause removal. Even with shuffle, debias still adds ~+0.8 on v2, so both used in final eval.

2. Training & Data

2.1 Streaming Data Loading

CJ decision replay data 41 GB. Initial script loaded entire file into memory per rank, plus another copy for length calculation → each process 42.9 GB, machine froze. Fix: build per‑file byte‑offset index (saved as .npy, memmap), seek to required line at training time. DataLoader workers share index; memory footprint tiny. Lesson: once data exceeds a few GB, never "load all first"; check per‑process duplication.

2.2 Val Acc Highest but NLL Worst

At step 800: accuracy 0.7884 (peak), ECE 0.032, Brier 0.2859 (all best), but hard NLL 1.045 (≈2× step 600’s 0.581). Cause: NLL = -log p(correct) unbounded; a few extremely confident wrong predictions (p≈1e‑10 → penalty 23) spike average NLL while accuracy barely moves. Training log showed grad_norm exploded from 7–15 to 698 at step 790–1030 ; gradient clipping capped step size but direction dominated by outliers. Step 800 validation happened 10 steps into explosion.

Metrics table:

step  accuracy  ECE     Brier   hard NLL
600   0.7796    0.054   0.2930  0.581
800   0.7884    0.032   0.2859  1.045
1200  0.7839    0.042   0.2974  0.602

Lessons:

Checkpoint selection cannot rely solely on accuracy; decision models output probabilities for downstream thresholding — NLL and ECE better reflect calibration. Author picks best by distributional NLL.

"Highest acc" often just noise: steps 600–1587 accuracy range 0.78–0.79, difference <25 questions on 2,836‑item val set (std err ±0.8%).

Always log grad_norm; when NLL jumps, check gradients first.

2.3 Collapse from Label‑Sorted Data

Final step (1587) overall score higher than step 600, but two benchmarks collapsed: RAGTruth (hallucination detection) 48.9 → 2.0 (98% "has hallucination"); Amazon ESCI (product relevance) 24.1 → 0 (99% class S). Lost 3.4 index points (27% of gap to top‑5 4B). Root cause: training files sorted by label .

RAGTruth: first 60% "no hallucination", last 40% "has hallucination".

ESCI: blocks ordered C → E → I → S.

Training read files sequentially with only 256‑line shuffle buffer — unable to break large‑block ordering. Last steps saw only single‑label batches. Scanning 869 shards: 9 sources, ~53k rows (5.6% of mix) affected. Step 790 grad explosion likely also from single‑label chunks.

Code preserved order in two places: (1) label balancing concatenated groups by label name; (2) sources under size limit kept original file order (already label‑sorted). Fix: globally shuffle each source’s lines before writing; add unit test checking label proportion in first/last chunks of output.

Lessons:

Never trust raw data order. Global shuffle at construction + streaming buffer shuffle at training — both layers needed.

Val metrics must be per‑source. Author’s val set already showed ESCI acc 17% but missed because only overall metrics watched; RAGTruth only 18 samples in val. Now require ≥200 balanced samples per source, log per‑source metrics, select checkpoint by macro‑average.

2B training also had single‑source drop (product classification 75% → 30% at step 3000, recovered at 4000). Invisible in aggregate, obvious per‑source.

2.4 Shortcut Learning in Data

Models exploit spurious correlations. Removed during mix construction (63→76 sources, 1M+ questions):

Phishing email era shortcut : phishing from 2023, normal from 2002 — model uses year. Also fixed domains, list footers. Dropped worst sources; scrubbed features from others.

Cross‑task leakage under bidirectional attention : GSM8K had 4‑option and 10‑option versions sharing same answer, packed in same sequence. Bidirectional attention lets one question see other’s candidates → trivial answer. Fix: don’t pack GSM8K. This bug is unique to bidirectional + multi‑question packing ; causal models immune.

Test‑set deduplication : all training data deduplicated against leaderboard’s 140k requests using 13‑char (Chinese per‑char) overlap; only train splits used; no Jev model outputs as labels.

2.5 Catastrophic Forgetting in Small‑Data Fine‑Tuning

Feasibility experiment: few‑thousand extraction samples, batch 16, lr 1e‑4 → Ifeng news classification dropped 86.5% → 79.0%, NLL 0.35 → 0.92. Control (classification‑only) also dropped, so issue not extraction task but fine‑tuning recipe. Original decision model trained with lr 1e‑4, global batch 1024. Same lr with batch 16 → per‑sample update strength ~64× higher ; 600 replay samples insufficient. Lesson: when continuing training on a converged model, either lower lr or use gradient accumulation to restore original batch magnitude.

2.6 Three Disjoint Data Splits

Decision models output probabilities → need temperature calibration. Strict separation:

Checkpoint selection: val subset (hash partitions 1–4).

Temperature fitting: another val subset (partition 0).

CJ‑Bench test set: only read once at final report, never during training/selection.

Easy to accidentally peek at test set for checkpoint picking or fit temperature on same data used for selection, inflating calibration metrics.

3. Classification & Extraction Coexistence

3.1 Unified Model Design

Classification and extraction share backbone + LoRA; only readout heads separate.

Classification: read candidate distribution at last position.

Extraction: final hidden states → tiny span head (~3M params on 0.8B, <0.4%), per‑candidate softmax over [non‑entity, type1, type2…].

Merge LoRA, write span head tensors into same safetensors; single weight release. Request field questions → classification; types → extraction; both → single forward returns both (2B short input: combined latency 47 ms vs 95 ms separate).

Design tip: Echo layout . Qwen is causal; native tokenization splits 9–14% entity boundaries (e.g., "是三" "集团的" in one token). Solution: tokenize text normally, then repeat character‑by‑character; spans taken from repeated copy. Benefits: every character aligns to token boundary; repeated copy sees full right context, giving causal model bidirectional‑like view. This change alone: CLUENER +7.7 F1, People’s Daily +5.2 F1.

3.2 Joint Training Forgetting

First joint run on CJ‑IE (Chinese extraction multi‑task) with 0.8B: extraction good, but decision capability plummeted:

Original  Joint     Δ
All     83.11     71.02    −12.09
Legal   67.84     39.98    −27.86
Finance 74.00     53.81    −20.19

Four simultaneous causes:

No old‑task replay (only used for eval).

Per‑step update too strong (30–60 samples/step).

Extraction gradients modify shared params.

Checkpoint selection ignored old‑task metrics; trained to end.

4B similar, stopped mid‑way. Fixes applied:

Old‑task replay : stratified sample from CJ train, full legal/finance (worst drops) included.

KL anchor : on replay samples add KL(original ‖ current). Original model obtained via PEFT disable_adapter() (turns off LoRA).

Gradient accumulation : ~400 samples per update, lower per‑step strength.

Checkpoint by old capability : evaluate at step 0 baseline; only consider checkpoints where old‑task acc drop ≤1 point, then pick best new‑task.

With replay + KL, v2 CJ‑Bench back to 83.25; v3 joint from base with ~13M decision replay → 83.46 (slightly above original). Unified 4B later reached 86.86, +0.87 over pure decision 4B.

3.3 Retention Tax on Extraction

Forgetting fixed, but extraction degraded: extraction‑only (decision forgotten) F1 73.9 vs unified 0.8B 63.1 — a 9.5 F1 gap after optimal threshold tuning. Multiple factors changed simultaneously (init, replay, KL, RLCD, batch, loss weight); controlled experiments queued to isolate. Lesson: "can coexist" ≠ "no mutual harm". Report unified model with single‑task baselines to quantify trade‑off.

3.4 Extraction Threshold: Calibrated Probability ≠ Optimal Decision Threshold

Training: "non‑entity" spans dominate → weight 0.2 for none class; inference: add log 5 to none logit to restore prior → output probabilities calibrated. But using calibrated probability directly for entity decision yields high threshold: recall 15–20 points below precision. Dev sweep found decision shift 0.75 optimal ; all three sizes gain 1.4–1.8 F1, recall +6–9 points.

Key: separate calibration from decision . Reported probability keeps log 5 (calibrated); entity emission uses separate decision_shift (0.75). If none_shift altered, F1 rises but probabilities lose calibration.

3.5 Extraction Generalization to Unseen Types

Early experiment: trained on CLUENER + People’s Daily (13 types). Zero‑shot on Resume NER: person 100, org 78, title 69, but nationality, education, major, ethnicity, location all 0 . Span head learned detectors for seen types, not "read type description then match". CalOn achieves zero‑shot via thousands of type names in training.

Improvements:

Add type‑rich data (B2NERD hundreds of Chinese types); CJ‑IE now covers 35 sources.

Shuffle/replace/anonymize type names during training to force reading descriptions.

For custom business types, few‑shot labeled data far outperforms prompt engineering : intent+slot experiment — 0 synthetic → 15.2 F1, 100 → 76.9, 500 → 96.4, 2000 plateau.

4. Real User Feedback: Why "Parking Management Fee" Not Extracted?

User needed simultaneous classification (attitude, intent) and extraction ("customer address", "consulted business") in property‑management dialogue. "Parking management fee" missed. Large 30B‑A3B model worked but 500+ ms latency.

Diagnosis

Vague definition : "business customer consulted" — model unsure what noun to pull. Another example: "which part of property fee owed" mixed general/specific; "parking management fee" prob 0.16 < non‑entity. Sharpened definition ("only fee item names, e.g., parking management fee, elevator fee; exclude 'how much', 'how to pay'") fixed it.

Type interference : two types in one request — address correct → business missed; business correct → address missed. Separate requests: address 7/7, business 6/7. Even swapping type order dropped address 3/6 → 1/6.

Echoing input text in response : wasteful; just disable in wrapper.

Why Interference?

Prompt lists type definitions sequentially; model reads left‑to‑right. Each character’s representation in "echo" section conditioned on all preceding type definitions. Adding "consulted business" changes representation of "熊" (surname) → its score as "customer address" shifts. Order similarly matters. Common types (person, location, org) have high scores (~0.9) and are stable; custom types near 0.5 are fragile.

Solutions

Prefer choice over extraction when candidates enumerable : fee items as choice question → "parking management fee" prob 0.94, far more stable. Extraction for open‑ended candidates.

Split types into separate requests : two async calls ≈130 ms total; or add "type groups" in API for independent per‑group scoring.

Reduce non‑entity bias : change none prior from ×5 to ×2 → both types correct 4/9 → 8/9, no new false positives.

Root fix: fine‑tune on few hundred annotated dialogues with custom types to make them familiar.

Latency: unified model classification+extraction ~47 ms (2B, short dialogue, A800) — 10× faster than 30B‑A3B, core value of small decision models.

5. Engineering Pitfalls

Batch=1 padding leak : server pads single input to 64‑multiple left. Qwen3.5 linear attention only zeros padding when batch≥2 → pad tokens entered recurrent state, probability offset up to 0.11. Batch eval unaffected, so missed in testing. Fix: no padding for batch=1; add regression test.

4‑GPU NCCL hang : without NCCL_P2P_DISABLE=1, 4‑GPU training stalls (CPU 100%, GPU 100%, only weights in VRAM, no progress). Cause: cross‑NUMA PCIe P2P deadlock on this machine; 2‑GPU fine.

pkill -f / pgrep -f self‑sabotage : SSH running pkill -f <train_args> matches its own shell command line → connection drops. pgrep -f waiting for previous task matched the SSH shell that launched it → waited 20 hours.

Editable install shadows new code : shared venv had old decision package installed editable; scripts from other directories silently imported old version. Fix: export PYTHONPATH=$PWD before running.

Variable name collision : training script with --rlcd assigned loss to out, but out was also output directory → crash at first checkpoint save. Smoke test must cover "save + resume".

Checkpoints overwritten in place : script keeps only last and best, overwriting each time. Step 800 NLL anomaly checkpoint lost, couldn’t per‑sample audit. Now freeze extra step‑N checkpoints periodically.

Custom heads not GGUF‑compatible : llama.cpp only recognizes standard LM head; span head and pointer head unreadable → unified and pointer models run only in PyTorch.

6. Community Q&A

Q1: RLCD Details

TypeSafe published RLCD objective but no reproducible loss. Author implemented based on public reproductions (Chinese‑Jev etc.) in decision/losses.py.

One‑liner: add Gaussian noise to logits, measure score of perturbed distribution, push logits toward higher‑score directions.

Formula:

p      = softmax(z)                          # z: logits per candidate
loss   = CE(t, p) + λ · L_rlcd               # t: target distribution (one‑hot for hard)
L_rlcd = -E_ε[ S(softmax(z + σε), t) ]
S(q,t) = Σ t_k · log q_k                     # log score
       + 0.75 · <t, q> / ||q||              # spherical score
       - RPS(q, t)                           # ranked probability score (score tasks only)

S : combination of strictly proper scoring rules — only maximized when q = t; prevents "always 99%" gaming. Log score overall; spherical tolerant to extremes; RPS handles ordered tasks (guess 3 when true 4 penalized less than guess 1).

σε : Gaussian noise σ annealed 0.4→0.1, projected to sum‑zero subspace (softmax invariant to constant shift).

Gradient : REINFORCE — treat z as Gaussian policy mean, sample G perturbations as actions, compute reward R_g = S(…), baseline = mean of other rewards, advantage A_g, backprop -A_g · log N(z̃_g; z, σ²I). Intuition: push logits toward perturbations that improve score. Presets: G=4, log‑score floor log(1e‑4), per‑batch advantage normalization.

Two caveats:

CE is already strictly proper; RLCD adds smoothing and ordered‑task care, not essential for learning probabilities.

RLCD isolated gain not yet measured: 0.8B no RLCD, 2B/4B with RLCD but also size/batch changes. Need controlled ablation.

Q2: Using JEV in Agent Loop Harness

Not yet in production; thoughts from Decision Studio "agent router" scenario.

Core view: JEV does not "think", it "judges". No generation, no planning; gives calibrated probabilities in tens of ms for "should I?" / "which one?". Fits as "fast thinking" layer before LLM at every agent branch point.

Tool selection / routing : each step ask "which tool to call" (choice). >26 tools → pointer head single pass.

Decide whether to ask clarifying question : "missing key info?" (noul) → ask first, don’t rush to call.

Risk‑based thresholds : "execution risk level" (score). Studio defaults: read‑only (weather) 0.4, write (send msg, create calendar) 0.65, financial (transfer) 0.85; below → user confirm or human handoff. Requires calibrated probabilities — hence NLL/ECE emphasis.

Post‑tool verification : retrieval relevance, hallucination detection (like RAGTruth), task completion (stop condition).

Multiple judgments in one forward : same agent state → several questions (tool, missing info, risk) packed in one sequence; v2 multi‑question readout built for this.

Caveats:

Agent context often long; decision model context limited (e.g., ToolRet 27k tokens too large). Feed compressed state, not full trajectory.

Thresholds must be calibrated on own data; above numbers just starting points.

Low confidence needs fallback: hand to LLM for deep reasoning or human. JEV’s value: filter easy judgments, flag uncertain ones.

Closing Summary

Finalize readout structure early. Category ceiling, bidirectional attention — must be present from step 1; retrofitting costly.

Data fails more often than models. Ordering, shortcuts, leakage — each can make model "look good" or "suddenly collapse". Global shuffle, test deduplication, per‑source review.

Validation must be large, per‑source, probability‑aware. Overall accuracy misses collapses and picks wrong checkpoints.

Quantify multi‑task coexistence cost. Forgetting controllable via replay + KL + old‑capability checkpointing, but extraction side loss must be measured separately.

Prefer choice over extraction when candidates enumerable. Choice far more stable than open extraction.

Engineering details silently change results. Padding, mask, env vars — bugs don’t crash, just quietly degrade metrics.