Mastering Few-Shot Prompting: Principles, Bias Fixes, and Example Design

Few-shot prompting provides a small number of task examples inside the prompt to help the model understand the task and improve performance on similar tasks. These examples act as guidance, showing the model how to handle and respond to specific types of tasks or questions.

For sentiment classification of movie reviews, two labeled examples are given:

Review 1: "This movie is a complete waste of time." Sentiment: Negative. Review 2: "I couldn't stop laughing throughout the whole movie!" Sentiment: Positive.

When presented with a third review, the model predicts:

Review 3: "The special effects are great, but the plot is confusing." Sentiment: Neutral (or mixed).

Improved performance: the model better understands the task, producing more accurate outputs.

Rapid adaptation: swapping examples quickly changes the task type or domain.

Reduced data requirement: only 2‑5 examples are needed, eliminating the need for large labeled datasets or fine‑tuning.

The paper *Language Models are Few-Shot Learners* notes that larger models can more efficiently use contextual examples without fine‑tuning. As the number of examples (K) increases, performance improves, especially for larger models.

Performance rises with more natural‑language task descriptions and more examples in the context.

Few‑shot learning improves significantly as model size grows.

Principles of Few-Shot Prompting

A clear and logical prompt structure reduces the model's comprehension cost. A standard structure should contain three parts: task description, example set, and the problem to solve.

Task description should be concise and explicit: use plain language to state the goal and output format, e.g., "Classify news headlines as 'Tech' or 'Sports' without additional explanation."

Example ordering should be logical: arrange from simple to complex or from core to edge cases, helping the model build understanding step by step.

Add brief explanations to enhance interpretability: for complex tasks, append a short rationale after each example output.

Examples are the core of few-shot prompting; their quality directly determines the model's task comprehension. They should follow three principles: typicality, consistency, and diversity.

Typicality first: choose examples that cover the core scenarios of the task.

Absolute format consistency: keep the "input‑output" structure identical across all examples.

Diversity coverage: ensure examples represent the main variants of the task, balancing label distribution.

Control quantity and length: 3‑5 examples are optimal; keep each example concise.

Factors Affecting Example Quality

Designing examples involves multiple factors that impact model performance, especially given limited context windows in early LLMs.

Example quantity – Adding more examples generally improves performance, though benefits diminish after about 20 examples for many models.

Example ordering – The sequence can cause accuracy to vary widely, from below 50% to above 90%.

Label distribution – Imbalanced label counts can bias the model toward the majority class.

Label quality – While some studies suggest noisy labels may not drastically hurt large models, accurate labeling often yields better results.

Example format – Common formats like "Q: {input} A: {label}" work well, but the optimal format may differ per task.

Example similarity – Choosing examples similar to the test input usually helps, though diverse examples can sometimes be beneficial.

Overall, designers must consider quantity, order, label distribution, quality, format, and similarity to minimize bias and maximize performance.

Eliminating Model Bias

The paper *Calibrate Before Use: Improving Few-Shot Performance of Language Models* identifies three biases that can affect model outputs:

Label frequency bias: the model tends to pick the label that appears most often in the prompt.

Recency bias: the model favors the answer appearing last in the prompt.

Common‑word bias: the model leans toward high‑frequency words seen during pre‑training.

To mitigate these biases, a simple calibration procedure is proposed: first, replace the test input with meaningless content (e.g., "N/A" or "[MASK]") and observe the model's predictions to measure bias; second, adjust the model's output distribution so that the meaningless input yields a uniform prediction (e.g., 50% positive, 50% negative), thereby neutralizing the bias.

Constructing More Examples

With larger context windows (e.g., Gemini 1.5 Pro handling up to 1 million tokens), hundreds or thousands of examples can be included. The paper *Many-Shot In-Context Learning* validates the benefits of many examples but also raises the challenge of constructing them.

Two recent methods for generating examples are:

Reinforced ICL: let the model generate solution steps, then select the correct steps as examples, often outperforming manually written ones.

Unsupervised ICL: provide only the questions (no answers) and a brief output format instruction; the model can still learn effectively, sometimes surpassing few‑shot with human explanations.