LemonDecider 4B Beats Jev on Decision Index, Fully Open-Sources Weights & Training Data

When building agents, search systems, or customer-service pipelines, developers often need a model that can make structured judgments — classify intent, check if evidence is sufficient, or pick the right tool — rather than generate long free-form answers. LemonDecider 4B is designed exactly for this: given background context and a list of questions with candidate options, it returns a direct, program-readable decision (choice, yes/no, or score) without requiring post-hoc parsing.

Benchmark Results

On the Decision Index 0.4.1 Public benchmark, the 4B-parameter LemonDecider achieves 61.67 points, edging out Jev 1.14 (60.7). Median inference latency across the full ~140k evaluation set is only 50.7 ms, making it practical for low-latency production use.

Open-Source Scope

The team open-sources the complete artifact chain:

Model weights – Hugging Face repository: inclusionAI/LemonDecider Training data (Decision Mix) – 1,199,692 independent samples sourced from 287 upstream datasets covering 864 task labels. Dataset: inclusionAI/LemonDecider-Decision-Mix Training & evaluation code – Scripts to reproduce training from the base model and to run the Decision Index evaluation. Results log: inclusionAI/lemondecider-decision-index-results According to the authors, LemonDecider is the first model on the Decision Index leaderboard that beats Jev 1.14 while publicly releasing its entire decision-training corpus.

Interface Design: Three Output Modes

LemonDecider accepts a context and one or more questions (each with candidates or a rating scale) and returns structured results in a single forward pass when length permits:

choice – Select from a candidate list (e.g., “Which tool should handle this request?”)

noul – Binary yes/no judgment (e.g., “Is the current evidence sufficient to answer?”)

score – Rate on a given scale (e.g., “How relevant is this retrieved passage?”)

Multiple questions sharing the same context can be batched together, avoiding repeated context encoding.

Training Data Construction Principles

1. Preserve Original Supervision Signals

Instead of re-labeling with a larger model, the team kept original annotations. About 61,057 samples (≈5%) retain soft labels — e.g., if 70% of annotators chose A and 30% chose B, that distribution is preserved rather than collapsed to a single hard label.

2. Control Source Proportions & Quality

Per-source caps prevent a few large datasets from dominating. Additional hygiene steps include cross-source deduplication, evaluation-sample fingerprint checks, answer-leakage detection, and input-length filtering.

Data Format & Licensing Notes

Each public row contains id, state, question, target, label — one question per row. Original multi-question groupings are not retained; users must design their own grouping for multi-turn judgment tasks. The accompanying row_licenses.jsonl tags each row’s license: ~66.2% commercial-use, 31% unclear, 2.8% non-commercial only. Downstream users must verify upstream terms.

Task Coverage

The 864 task labels fall into four practical categories:

Understanding Requests – Identify user intent, route to tool/service

Checking Information – Judge if evidence supports a conclusion, detect unsupported claims

Filtering Content – Assess retrieval relevance, sentiment, safety risks, harmful content

Handling Details – Aspect-level sentiment, next-step selection from candidates

Version 3 Improvements

Beyond scaling data from ~626k to ~1.2M samples, v3 introduces multi-question joint training, checkpoint weight averaging, and temperature calibration. The authors emphasize that the gain is not merely from more data.

Getting Started

Local inference (example):

CUDA_VISIBLE_DEVICES=0 python code/infer.py --input examples/request.json

HTTP service: CUDA_VISIBLE_DEVICES=0 bash code/serve.sh Smoke test: python code/smoke_test.py The model uses custom code; see the repository for environment details. Context window is 8192 tokens (including template). Requests exceeding this are auto-chunked — each chunk keeps the full background and options, results are merged. If a single “background + one question” exceeds the limit, the service errors explicitly rather than truncating silently. Chunking implies multiple forward passes, so not all requests complete in one pass.

Limitations & Future Work

LemonDecider is not a general-purpose chat model; it targets scenarios where both the question and candidate set are well-defined. The team plans to improve data quality, update models, and refine reproduction docs. They invite real-world testing: which judgments work, which fail, and how performance shifts with domain-specific data.