ShoppingBench: AAAI'26 Benchmark for Shopping Agents in Real‑World E‑Commerce

Background

Existing e‑commerce agents focus on simple intents such as product search and purchase. To address more complex user goals—coupon usage, budget management, and multi‑product sellers—the authors propose ShoppingBench, an end‑to‑end benchmark covering realistic, intent‑grounded scenarios.

They construct a virtual shopping environment with over 2.5 million real products and evaluate 17 large language models (LLMs). Even the best model, GPT‑4.1, achieves less than 50 % absolute success rate, highlighting the benchmark’s difficulty.

Dataset Construction

Virtual Environment

Two optimization paradigms are discussed: Parameter‑Driven (model updates via SFT or RL) and Parameter‑Free (prompt engineering, tool augmentation, RAG, multi‑agent coordination). The authors adopt the ORM‑Virtual paradigm, which offers better controllability and scalability than PRM or ORM‑Realistic.

The environment includes structured product data (some attributes extracted with Qwen2.5‑VL‑72B‑Instruct), a Pyserini‑based search engine, and a standardized toolset (product search, detail view, web search, Python executor). It supports mainstream LLMs such as gpt‑4.1 and Claude‑4‑Sonnet via a ReAct‑style agent framework.

Scalable Data Synthesis

Products Finder: locate items based on attribute descriptions.

Knowledge: answer questions requiring external knowledge; built from SimpleQA with 4 000+ Q‑A pairs, filtered to 310 valid pairs.

Multi‑products Seller: find a single store offering all requested items.

Coupon & Budget: parse coupon rules and recommend optimal product bundles under budget constraints. A five‑step pipeline randomly selects coupon types, items, core attributes, generates coupon parameters, and synthesizes user commands using GPT‑4.1.

The synthesized commands and tool‑call trajectories can be validated automatically, enabling low‑cost, scalable batch generation.

Method

Evaluation Metrics

Two core metrics are defined for each intent:

Cumulative Average of product Relevance (CAR): measures match between recommended product attributes and user‑specified attributes.

Absolute Success Rate (ASR): proportion of samples where CAR and all constraints (e.g., coupon‑after price < budget) are fully satisfied.

Trajectory Distillation

Using GPT‑4.1, tool‑call trajectories are generated for 2 410 user commands. Low‑quality trajectories are filtered out via rejection sampling, keeping only those with ASR = 1.

Distillation + Reinforcement Learning

From each trajectory, multiple steps are sampled, yielding a training set of 5 552 steps. Inputs consist of user commands and observations; outputs include reasoning traces and the next tool action. Supervised fine‑tuning (SFT) on Qwen3‑4B improves handling of complex commands and multi‑turn reasoning.

Subsequently, GRPO reinforcement learning with a reward composed of format reward and tool‑calling reward further enhances tool usage. The tool reward measures name, parameter, and value match between the model’s action and the ground‑truth tool call.

Experimental Results

After SFT + GRPO training, the distilled agent surpasses GPT‑4.1 on three intents—Single‑product selection, Multi‑product seller, and Coupon & Budget—while lagging on the Knowledge intent that heavily relies on external knowledge.

Ablation studies on Think vs. No‑Think modes reveal that removing the Think step improves simple tasks (single‑product) but harms complex tasks requiring logical or mathematical reasoning. The authors suggest future post‑training to enable dynamic mode switching.

Application

The previous LazzieChat assistant used a Workflow mode that could not cover complex e‑commerce intents. Leveraging ShoppingBench, the team built an Agentic mode based on Qwen3‑30B‑A3B, which adaptively decides actions and tool calls, delivering noticeable performance gains for complex queries.

Conclusion

ShoppingBench provides a realistic, intent‑grounded benchmark for e‑commerce agents, featuring a 2.5 million‑product virtual sandbox, scalable data synthesis, trajectory distillation, and reinforcement learning. The framework enables a lightweight model to match or exceed GPT‑4.1 on multiple tasks and has been successfully deployed in LazzieChat, advancing intelligent, efficient e‑commerce interactions.