LongDS v1.1 Benchmark Released: GPT-6 Astra Tops Long-Horizon Data Analysis Lite Leaderboard

Introduction

LongDS-Bench is a new benchmark for evaluating agents on long-horizon, multi-turn data analysis tasks. Developed by Zhejiang University, Ant Group, and their joint lab, the paper has been accepted at EMNLP 2026. The benchmark addresses a critical gap: real-world data analysis involves dozens of turns where each request depends on correctly maintaining and evolving analysis state — data scopes, metric definitions, hypotheses, and intermediate tables — across the conversation.

What Is LongDS?

LongDS evaluates whether an agent can consistently use the correct analysis state throughout a prolonged dialogue. The full benchmark contains 68 tasks and 2,225 interaction turns sourced from real Kaggle analysis notebooks, spanning six domains: business, community, education, geoscience, social good, and sports. Each task averages 32.7 turns ; each turn directly depends on approximately 2.85 historical turns , with a maximum dependency distance averaging 11.3 turns .

The benchmark focuses on analysis state maintenance and evolution , including:

Reusing existing rules

Modifying default definitions

Trying temporary hypotheses

Retrieving and combining historical versions

Real-World Motivation: Analysis Gets Harder the Longer It Runs

Typical data analysis starts simple — read a table, compute averages, find top items. But users then narrow samples, adjust metric weights, compare hypotheses, and re-check earlier conclusions. As the analysis progresses, the computational basis changes: the same phrase "recompute ranking" may require different data subsets, formulas, and intermediate tables at different turns. An agent must distinguish which modifications persist, which are one-off comparisons, and which old results remain relevant.

Crucially, code can execute successfully yet use the wrong version of data or metrics , producing silently incorrect results that propagate forward. LongDS formalizes these dependencies as "analysis state" and tests whether agents can preserve and apply the correct state over the long term.

How LongDS Differs from Existing Benchmarks

Existing data-analysis benchmarks either provide a complete goal upfront or use multi-turn dialogue to clarify a single initial task. LongDS is unique because every turn introduces a new analysis request with its own reference answer , and answering later turns requires inheriting, modifying, or restoring previously established state.

The difficulty lies in multiple valid state versions coexisting; the latest version is not always the one needed. For example, a user may ask to adopt a new metric definition while reverting to an earlier data scope — the agent must understand when each modification took effect, which calculations it affects, and how to combine them correctly.

State Evolution Patterns

LongDS categorizes state behaviors into six patterns (illustrated with a "user activity" example):

Initial Construction — establish new state (data scope, metric, intermediate table)

Update — modify an existing state element (e.g., change metric formula)

Rollback — revert to a prior version of a state element

Temporary Hypothesis — apply a change only for the current comparison, preserving the default

Reference — explicitly retrieve a specific historical result

Composition — combine multiple historical states (e.g., use current scores with pre-penalty scores and baseline)

Inheritance of valid state is the default between turns and is not counted separately. A single turn can involve multiple behaviors simultaneously.

From Real Notebooks to Verifiable Multi-Turn Tasks

4.1 Construction Pipeline

Collected 256 notebooks from 64 Kaggle competitions/public datasets.

Filtered for data availability, code executability, and analytical depth → retained 77 executable notebooks .

Selected three notebooks, preserved their main analysis thread, organized code cells into analytical segments, identified reusable data scopes/metrics/intermediate results, and manually built three multi-turn seed tasks.

Extracted a "task construction skill" from these examples and used Codex to assist building the remaining initial tasks. Each turn includes user request, executable reference code, reference answer, and state/dependency annotations.

Initial tasks underwent expert review, agent-assisted validation, and final consistency checks. Reference code was re-executed to verify dependency necessity, question clarity, and answer reliability; unfixable tasks were removed.

Redundant prompts that merely repeated early rules were removed (while preserving cross-turn state-tracking requirements) to keep tasks concise.

4.2 Task Example: 36-Turn Netflix Market Opportunity Analysis

The paper walks through a concrete 36-turn task to illustrate state complexity:

Turn 1: Clean program catalog and search-trend data, build analysis tables for later use.

Turn 2: Compute market opportunity scores using specified time windows and country attribution rules.

Turn 3: Explain why top-ranked markets received their scores — must reuse Turn 2's candidate markets and weights.

Turn 18: Temporarily lower the long-movie duration threshold from 110 to 100 minutes to see which markets stay in the top five. The agent must recompute on the long-movie analysis basis while preserving the original default scores for subsequent turns.

Turn 24: Requires combining results from three different turns: select top eight markets by Turn 23's current scores, retrieve Turn 21's scores (before director penalty), and compare with Turn 12's baseline — all while keeping the current default scores unchanged.

This demonstrates the need to disambiguate temporary hypotheses, historical versions, and current default state, and to apply them exactly as requested.

v1.1 Updates and the Lite Subset

Following EMNLP 2026 acceptance, the team released LongDS v1.1 and v1.1-Lite .

v1.1 Changes

Re-audited the entire benchmark.

Revised ambiguous task phrasing.

Corrected errors in reference answers.

Clarified tie-handling rules and output requirements.

These revisions preserve the original analytical logic and cross-turn dependencies while enabling more accurate assessment of whether agents use the correct analysis state.

v1.1-Lite

To facilitate repeated testing, the team selected 24 tasks from v1.1 that best differentiate agent performance. The Lite subset:

Covers all six domains and multiple state-evolution behaviors.

Retains full content and all turns for each selected task — total 777 turns , average ~32 turns per task.

Allows developers to evaluate long-horizon state management with less time and compute cost.

LongDS supports five runners besides DSGym: Codex, Claude Code, Kimi Code, and Qoder , all with local and Docker modes. In Docker mode, each task runs in an isolated container, and the runner uses native session continuation so all turns of a task execute in a single continuous agent conversation.

Leaderboard Results (v1.1-Lite, as of Sep 7, 2026)

Evaluated with DeepSeek-V4-Pro as judge (semantic + numeric match, 0/1 per turn, then per-task average accuracy, then macro-average across tasks):

GPT-6 Astra: 78.17

Claude Fable 5.1: 76.53 (1.64 points behind)

GPT-5.6-sol (Codex): 70.71 (7.46 points behind Astra)

Other models show larger gaps.

Note: The original v1 experiments used DSGym and different models (Gemini-3.1-Pro 48.45, GPT-5.4 43.50, Claude-4.6-Sonnet 41.56). Scores across versions are not directly comparable due to changes in models, task versions, task sets, and evaluation frameworks.

Error Analysis from Original v1 Experiments

8.1 Performance Degrades Sharply Toward Task End

Aligning tasks by relative progress (first 10% → last 10%), overall accuracy drops ~46.8 percentage points . Higher historical dependency (more direct dependencies, longer lookback distance) correlates with lower accuracy. Rollback requests are harder than initial construction tasks. This shows that step-wise computational difficulty alone cannot explain performance; the burden of retrieving and validating the correct historical context grows with analysis length.

8.2 Error Taxonomy: Cascading Errors Dominate

Analyzing 3,207 error turns from sampled tasks, three long-horizon error types account for 52%–69% of each model's errors:

Context Memory Errors — forgetting historical information (relatively rare).

State Management Errors — choosing the wrong version, updating the wrong object, or restoring an inappropriate historical state.

Cascading Errors — largest share . The current turn's logic may be correct, but it relies on an earlier erroneous intermediate state. The agent propagates that error forward, so inspecting only the current code fails to reveal the root cause.

Reducing these errors requires not just saving information but also verifying which version is currently applicable and whether retained intermediate results are correct.

8.3 Environment Reset: Double-Edged Sword

A diagnostic experiment with GPT-5.4 cleared the code environment (variables, intermediate results) at specified turns while preserving dialogue history. On originally weak tasks, reset yielded modest improvement; on strong tasks, accuracy dropped significantly. Reason: when existing state is wrong, rebuilding can reduce error propagation; when state is correct, wiping it discards needed results.

Additional observation: more interaction steps do not guarantee higher accuracy. If an agent persistently uses an erroneous state, extra reasoning or tool calls merely recompute the same wrong results.

The paper suggests promising directions: tagging analysis objects with provenance (source turn, valid version), separating temporary branches from default state, auditing key intermediate tables, and selectively recomputing only affected portions after detecting state conflicts.

How to Run Evaluation and Submit to Leaderboard

9.1 Download Data and Choose Runner

hf download zjunlp/LongDS --repo-type dataset --local-dir dataset

For Lite-only evaluation, follow the dataset page's targeted download instructions. Existing v1 data can be reused; only task files need updating.

Select a runner (Codex, Claude Code, Kimi Code, Qoder, or DSGym), set up the environment per the installation guide, and configure model access. All four native CLI runners support local and Docker modes.

9.2 Test One Turn, Then Run Full Lite

After preparing Python, Codex config, Docker image, and judge model credentials ( JUDGE_API_KEY, JUDGE_BASE_URL; default judge: deepseek-v4-pro), run a single turn of one task to verify the pipeline:

python runners/codex/run_codex_longds.py  --use-docker --task-limit 1 --turn-limit 1 --judge

Then run the full Lite subset (4 parallel tasks, sequential turns per task, persistent session):

python runners/codex/run_codex_longds.py  --use-docker --run-parallel 4 --judge

Adjust --run-parallel based on resource budget. For full v1.1, add --split full.

9.3 Verify Coverage, Keep Traces, Submit

Results are summarized in summary.json; task_avg_score × 100 gives the percentage macro-average. Per-turn responses, scores, and execution traces are saved in the experiment directory. Submit via GitHub Issue (https://github.com/zjunlp/DataMind/issues) or email (see repo), including benchmark version/subset, model and agent framework, run settings, task coverage, scores, and execution traces.