Jev-LDE: One Local Edit Boosts 1-Shot ICL Accuracy by 7 Points

Background: Demonstration Quality Matters in ICL

In-context learning (ICL) adapts LLMs to tasks by providing a few demonstrations in the prompt. However, simply retrieving the top‑k semantically similar examples (Semantic TopK) does not guarantee the best set for a specific target model. Irrelevant or redundant demonstrations waste context window and can hurt attention allocation. Model‑aware methods that repeatedly query the target LLM to evaluate candidate sets incur high inference cost, and the combinatorial space of choosing k from n candidates grows rapidly.

Key Observation: A Single Local Edit Can Yield Large Gains

Starting from a Semantic TopK set on TREC 4‑shot, the authors examined the one‑edit neighborhood (Keep, Delete one example, Replace one example with a candidate from the pool). An oracle that picks the best local edit using test labels improved accuracy by 26.0 percentage points for Llama‑3‑8B and 16.8 pp for Qwen2.5‑7B. A learned editor (Jev‑LDE) still gained 8.2 pp and 7.4 pp respectively, while random edits gave no benefit. This shows that a small local modification can correct many retrieval errors.

Method: Local Demonstration Editing (LDE) with Jev‑LDE

Framework

Pre‑selector : Any retriever (default Semantic TopK) produces an initial demonstration set D and a replacement candidate pool.

Editor (Jev‑LDE) : A small model (Qwen3‑1.7B) receives the task instruction, current query, D, and the candidate pool. It outputs a structured JSON action: Keep, Delete(i), or Replace(i, j).

Constraint : Exactly one membership edit per query. The edited set is then fed to the frozen target LLM for final prediction.

Training via Reinforcement Learning

No human annotation of "correct" edits is needed. For a given editing state, Jev‑LDE samples multiple actions; each resulting demonstration set is evaluated by the frozen target LLM (Meta‑Llama‑3‑8B‑Instruct during training). Reward = 1 if the target LLM predicts correctly, else 0. The policy is optimized with GRPO, allowing exploration over multiple valid actions without a single supervised label.

Experimental Setup

Training datasets : AGNews, TREC, DBPedia, GSM8K.

Held‑out datasets : Banking77, SST‑2, AQuA‑RAT, TACRED, TREC‑50.

Target LLMs : Qwen2.5‑7B‑Instruct, Meta‑Llama‑3‑8B‑Instruct, Qwen3‑8B, Qwen3.5‑9B, DeepSeek‑V4‑Flash.

Shot budgets : 1‑shot to 4‑shot (also 8‑shot and 16‑shot).

Baselines : Semantic TopK, BM25, TopK+ConE, LMS3, CASE.

Main Results: Classification Tasks

Overall Improvement

Across 48 classification configurations (3 datasets × 4 target LLMs × 4 shot settings), Jev‑LDE outperformed its Semantic TopK initialization in 46 cases and achieved the best or tied‑best accuracy in 44 cases . Average 1‑shot accuracy rose from 81.2% to 88.1% .

Low‑Budget Regime

Gains are most pronounced at 1‑shot. On TREC 1‑shot with Llama‑3‑8B, Jev‑LDE reached 65.6% vs. 52.8% for Keep. At 8‑shot and 16‑shot, Jev‑LDE still improved 18 of 24 configurations, though the margin narrows.

Transferability

Trained only with Llama‑3‑8B feedback, Jev‑LDE improved 35 of 36 configurations on Qwen‑series targets.

On held‑out Banking77: 14 of 16 configurations improved.

On held‑out SST‑2: all tested settings showed positive gains.

With DeepSeek‑V4‑Flash: TACRED +0.3–2.2 pp, TREC‑50 +2.4–4.4 pp.

Compatibility with Other Pre‑selectors

Using BM25, CASE, or TopK+ConE as the initial retriever, Jev‑LDE improved all 24 selector–target–shot combinations. Candidate pool size matters: on 1‑shot TREC, N=16 gave 80.6% accuracy, while N=32 dropped to 79.4% (larger decision space hurts).

Ablation Study (1‑shot TREC, Llama‑3‑8B Target)

Jev‑LDE: 65.6%

Keep original set: 52.8%

Llama‑3‑8B self‑edit (prompted): 57.8%

Untrained Qwen3‑1.7B: 31.2%

Random local edit: 23.2%

Jev‑LDE outperforms Keep by 12.8 pp and Llama self‑edit by 7.8 pp , confirming that RL post‑training is essential for effective action selection.

Inference Cost

On 1‑shot TREC with Qwen2.5‑7B‑Instruct: Semantic TopK wall time ≈ 82.1 s; Jev‑LDE ≈ 91.1 s ( +11% ), while accuracy jumps from 65.4% to 80.6%. TopK+ConE (which requires multiple target‑model evaluations) takes ≈ 197.5 s. Jev‑LDE’s overhead is comparable to lightweight baselines LMS3 and CASE.

Limitations

Task scope : Stable gains are observed on classification tasks. On math reasoning (GSM8K, AQuA‑RAT), differences are within ±1 pp, with slight drops (‑0.8 pp and ‑1.9 pp) on AQuA‑RAT for Qwen2.5‑7B and Llama‑3‑8B respectively.

Edit scope : Only one Keep/Delete/Replace operation within the candidate pool. If the optimal demonstration is absent from the pool or multiple edits are needed, LDE cannot recover.

Conclusion

Jev‑LDE demonstrates a simple yet effective paradigm: retrieve a decent initial set, then apply a single learned local edit. The 1.7B editor, trained via target‑model feedback with GRPO, transfers across unseen LLMs and datasets, adds only ~11% latency, and consistently lifts few‑shot classification accuracy. The approach trades a reduced search space for lower inference and learning cost, offering a practical post‑processing module for ICL demonstration selection.