Why Large Models Excel at Table Lookup Yet Fail at Future Prediction – Insights from TopBench

Implicit Predictive Table Question Answering

Traditional Table QA (TQA) assumes the answer exists in the table, so the model only needs to locate, aggregate, or verify facts. Implicit predictive TQA requires the model to infer unseen outcomes from historical patterns. The task is formalised as a two‑stage reasoning problem: (1) intent abstraction – extract the target column, feature profile, candidate set, constraints, and optimisation direction from the natural‑language query; (2) predictive inference – learn a mapping from the historical table and produce predictions for new profiles or states.

Four Task Categories

Single‑Point Prediction : predict a classification or regression label for a new profile not present in the table.

Decision Making : generate predictions for multiple candidates and select the best according to criteria such as “higher cost”.

Treatment Effect Analysis : perform what‑if reasoning by comparing predictions before and after a state change.

Ranking and Filtering : apply explicit or implicit filters, run batch predictions, and output a structured CSV of the top‑K results.

Examples of intent mapping: “budget for insurance bill” → regression on charges; “who will likely incur the highest cost” → predict for multiple candidates and pick the maximum; “will moving to another region lower my charges” → compare pre‑ and post‑change predictions; “top 3 females with largest projected payouts” → filter by gender, predict, then rank.

Benchmark Construction (TopBench)

TopBench collects 35 real‑world tables from Kaggle and other sources covering Healthcare, Finance, and Daily Consulting. Each table is paired with 779 high‑quality natural‑language queries, forming (query, CSV, ground‑truth JSON) triples. Task distribution: 274 Single‑Point, 186 Decision Making, 105 Treatment Effect, 214 Ranking/Filtering. Targets are balanced between regression (384) and classification (395). Table sizes range from <1,000 rows to >6 million rows.

Data construction uses a logic‑driven sampling strategy: hard‑negative pairs for decision tasks, noisy candidate pools for ranking, and a dual‑view question generation (user‑centric vs. data‑owner) to avoid template matching.

Evaluation Protocol

Because implicit predictive TQA mixes natural‑language reasoning with structured output, evaluation is split into two streams:

Natural‑language reasoning : an LLM‑as‑a‑Judge extracts final conclusions and intermediate predictions from the model’s answer, then verifies their presence via string, fuzzy, and logical matching.

Structured output (Ranking and Filtering): metrics include F1 for filter correctness, Set Recall, NDCG for ranking quality, and batch NMAE (normalised absolute error) for numeric predictions.

Regression‑type NL outputs are scored with a composite metric that combines point error, interval coverage, and a width‑penalty term to discourage overly broad confidence intervals.

Experimental Findings

TopBench evaluates general‑purpose LLMs, reasoning‑enhanced agents, and table‑specialised models under pure‑text and agentic (code‑execution) settings. Overall scores remain below 0.60; the best single‑point accuracy is around 0.65. The gap widens when models default to a “lookup” mindset instead of switching to a predictive mode.

Intent alignment is the first bottleneck: providing explicit target column, task type, and feature description improves many models (e.g., Qwen3‑Instruct single‑point rises from 0.43 to 0.56).

Even with correct intent, robust table modelling is the second bottleneck: a predict‑only baseline that receives gold‑standard structured inputs outperforms end‑to‑end agents (Single‑Point 0.66 → 0.76).

Tool usage matters: models that generate genuine predictive pipelines (Random Forest, Logistic Regression) outperform those that rely on pandas filtering or nearest‑neighbor heuristics.

Typical failure mode – “Exhaustive Retrieval Loop” – occurs when a model treats a future‑prediction query as a historical row search, iterating over rows until the context window is exhausted.

Analysis of Bottlenecks

7.1 Intent‑alignment bottleneck : Semantic ablation experiments show that adding target column, task type, and feature description to the prompt yields significant gains (e.g., Qwen3‑Instruct single‑point 0.43→0.56; DeepSeek‑V3.2 Treatment Effect 0.57→0.68; GPT‑5.2 single‑point 0.60→0.64). This demonstrates that many failures stem from mis‑identifying the task rather than from lack of coding ability.

7.2 Modelling bottleneck : A predict‑only baseline that receives gold‑standard intent and profile but uses a strong ensemble predictor still outperforms the best end‑to‑end agents (Single‑Point 0.66→0.76, Decision Making 0.65→0.72, Treatment Effect 0.65→0.69). Hence, even when intent is known, constructing an accurate predictive pipeline remains difficult.

Agentic workflows reveal differing tool preferences: DeepSeek more often imports scikit‑learn and trains models, whereas Qwen3 tends to use pandas filtering or nearest‑neighbor lookup. Generated code frequently defaults to simple algorithms (e.g., un‑tuned linear regression) that may be unsuitable for noisy, non‑linear data.

Typical Failure Example

The “Exhaustive Retrieval Loop” illustrates the core challenge: faced with a profile absent from the table, the model assumes the answer must be in some row, then exhaustively checks each row until the context window is exhausted. This behaviour reflects the strong prior in traditional TQA that “answers are in the table”, which must be overridden for implicit predictive tasks.

Resources

Paper: https://arxiv.org/abs/2604.28076

Code repository: https://github.com/LAMDA-Tabular/TopBench

Dataset on HuggingFace: https://huggingface.co/datasets/LAMDA-Tabular/TopBench