DexHoldem: HKU's Poker-Table Benchmark Unifies Robot Dexterity and Reasoning Evaluation

Hong Kong University and Yisheng Technology present DexHoldem , a benchmark accepted at the NeurIPS 2026 Workshop that evaluates embodied dexterous manipulation and agent perception/decision-making in a unified Texas Hold'em poker scenario. The benchmark moves beyond isolated skill testing by requiring the robot to pick up thin cards, read their faces, push/pull chips of four denominations, and maintain a clean tabletop — all while an agent observes the evolving game state and decides the next action.

Data Collection and Hardware

Researchers teleoperated a UR10e arm with a Shadow Hand to collect 1,470 successful demonstrations across 14 primitive actions : picking up, placing face-down, and revealing left/right hole cards, plus pushing and pulling four chip denominations. Each primitive has 105 demonstrations (100 training, 5 validation). The control policy receives three camera views (top-down, third-person, wrist) and joint states, outputting 30-DoF joint position targets at each step. A separate overhead camera feeds the perception agent.

Action Evaluation: TCR vs. SPSR

DexHoldem separates Task Completion Rate (TCR) — whether the goal object reaches its target — from Scene Preservation Success Rate (SPSR) — whether surrounding objects stay within allowed displacement, overlap, and position rules. A trial where the robot pushes the target chip but knocks another chip aside counts as TCR success but SPSR failure.

Nine policy architectures (including π₀, π₀.₅, RDT, ACT, Diffusion Policy, etc.) were each given 80 real-robot trials under identical training data and scoring. π₀.₅ achieved TCR 61.2% and SPSR 47.5% ; π₀ scored SPSR 47.5% . For π₀.₅, 38 of 80 trials succeeded on both metrics, while 11 completed the action but disturbed the scene — cases TCR alone would miss.

Training Data Scaling with RDT

Using RDT, the team varied demonstrations per primitive: 10, 20, 50, 100 . Compared to training the policy head from random initialization, a pretrained policy reduced action prediction error by 1.2%, 9.0%, 10.7%, and 11.3% respectively on validation data. The paper notes this improves offline prediction but real-robot reliability still requires physical testing.

Perception Agent Evaluation

The agent must infer eight game-state variables from a single overhead image: current step, active player, dealer button, community cards, chips in the betting zone, each player's remaining stack, and showdown result. 36 realistic scenes were constructed; 14 model+tool combinations (GPT, Claude, Gemini variants via Codex, Claude Code, Gemini CLI) were each tested three times per scene.

Two metrics: Avg (mean per-variable accuracy) and Overall (fraction of scenes where all eight variables are correct). Opus 5.5: Avg 80.6%, Overall 49.1% ; GPT 6 Astra: Avg 80.2%, Overall 48.1% . The gap shows models often get most variables right but fail on at least one critical item. Per-variable breakdown: GPT 6 Astra reached 93.8% on betting-zone chips and 100% on opponent stack , but only 20.5% on community cards ; Opus 5.5 also scored 20.5% on community cards, while GPT 5.5 led at 61.5% .

Full System Integration Test

A complete game loop was run with GPT 5.5 for perception/decision and π₀ for control . The system executes a primitive, re-observes to verify TCR and SPSR, then routes to the next primitive or retries. If the scene is corrupted, it requests human help.

Over 33 valid games , 4 completed fully (12.1%) . Of those, 3 required no human intervention , and 1 completed without any retry . The system triggered 34 retries ; 12 recovered a previously failed primitive . Critically, 3 of the 4 completed games relied on retries to progress, demonstrating that recovery behavior is essential for sustained interaction.

Significance

DexHoldem provides a concrete, reproducible testbed where manipulation fidelity, perception accuracy, and long-horizon decision-making are measured together. The reported numbers — TCR/SPSR gaps, data-scaling curves, per-variable perception weaknesses, and retry-dependent completion rates — give researchers specific failure modes to target in future work.