PhysCodeGym Benchmark Catches Physics Errors in Code Agents' 4D Video Reconstruction

PhysCodeGym: A Benchmark for Physically Valid 4D Reconstruction by Code Agents

Shanghai Jiao Tong University, together with Sun Yat-sen University and Carnegie Mellon University, introduces PhysCodeGym , a benchmark that evaluates code agents on their ability to reconstruct physical processes from real videos by writing executable Blender programs. Unlike prior benchmarks that only compare rendered frames, PhysCodeGym verifies the underlying physics through program and scene inspection, plus counterfactual simulations.

Benchmark Design

The benchmark comprises 156 tasks sourced from Pexels, Physics-IQ, BAAI-DataCube Physics-aware Videos, and IRIS. Each task provides an 8‑second, 30 FPS reference video (240 frames) covering four physical phenomena:

Rigid bodies (95 tasks) – e.g., domino chains, Newton’s cradle.

Fluids (40 tasks) – e.g., milk pouring into a glass.

Deformable bodies (12 tasks) – e.g., silk cloth draping a teapot.

Wind‑driven motion (9 tasks) – e.g., flags fluttering.

Agents start from an empty Blender scene and must write a Python script that creates geometry, materials, lighting, camera, and—crucially—physical properties (mass, friction, constraints, forces) so that the simulation reproduces the video. Each task allows 3 hours and 20 Blender launches for iterative trial‑and‑error.

Annotations and Evaluation Metrics

Every video comes with three annotation layers:

Object list – category, count, appearance, initial layout.

Event annotations – 578 total events describing which objects act, in what order, and the final state, with timestamps.

Physical criteria – human‑specified acceptable initial setups, driving methods, equivalent modeling choices, and the motion mechanisms, contacts, and support relations that must be checked.

Evaluation uses six metrics grouped into two scores:

Visual score (average of structural similarity, perceptual similarity, object accuracy).

Physical score (average of event score, motion score, physical validity).

Overall score – equal weight of visual and physical scores.

Physical validity is assessed by a verification agent that reads the submitted program and the native Blender scene, locates objects mentioned in the criteria, writes custom Blender/Python measurement scripts, and checks contacts (interpenetration), support relations, and dynamics (state changes over time).

Counterfactual Verification

Because the original video may not exercise all physical conditions (e.g., a ball never reaches the table edge), PhysCodeGym introduces counterfactual verification . The verification agent creates a scene copy, applies a controlled intervention—such as changing the release direction toward the table edge or removing a contact—and re‑runs the simulation. By comparing the original and counterfactual trajectories, it exposes hidden errors like incorrect support extents or missing collision dependencies.

Case Studies: How Physical Errors Are Caught

Newton’s Cradle: Energy Increases Instead of Dissipating

GPT‑6‑Astra reconstructed the cradle geometry and reproduced the swinging and collisions, achieving perfect object and event scores. However, energy measurement revealed a violation: after release, the total energy should decay due to friction and inelastic collisions. Fitting the free‑swing phase gave an initial total energy of 4.97 mJ , but by frame 236 the gravitational potential energy alone reached 9.44 mJ —about 1.9× the initial estimate.

Video 6: GPT‑6‑Astra Newton’s cradle energy check. The potential energy curve surpasses the initial total energy estimate (dashed line).

Billiard Ball: Floating Off the Table Edge

In a billiard task, the reconstructed ball rolled only on the table surface, so physical checks passed. When the verification agent changed the cue ball’s release direction toward the table edge, the ball moved past the visible table boundary yet remained at the same height—it was supported by an invisible extension of the table geometry. Restricting the support to the actual table surface made the ball fall correctly. The original trajectory never reached the edge, so the error was hidden until the counterfactual test.

Figure 2: Billiard ball after crossing the table edge. The candidate (left) stays suspended; the counterfactual with limited support (right) falls.

Experimental Results

Main Experiment: Nine Model‑Harness Configurations

Nine combinations of models (GPT‑6‑Astra, GPT‑5.6‑Sol, GPT‑6‑Luna, GPT‑5.6‑Luna, DeepSeek‑V4.1‑Flash, etc.) and harnesses (Codex, Claude Code, Kimi Code) were evaluated under a 3‑hour / 20‑launch budget. Tasks that failed to produce a rendered output received zero.

GPT‑6‑Astra led with overall 0.663 (visual 0.693, physical 0.633).

GPT‑5.6‑Sol ranked second at 0.570 overall.

Across all configurations, object accuracy consistently exceeded event scores, while motion matching remained weak (e.g., GPT‑6‑Astra motion score only 0.324). Model rankings shifted across dimensions: GPT‑6‑Luna had a slightly higher visual score than GPT‑5.6‑Luna but lower event and motion scores, indicating that visual fidelity does not guarantee dynamic correctness.

Six reconstructed cases with perfect object and event scores still exhibited energy anomalies, interpenetration, and support errors—further confirmed by counterfactual checks. This demonstrates that correct object and event reconstruction does not imply physically valid mechanisms .

Reasoning Setting Comparison (DeepSeek‑V4.1‑Flash on Claude Code)

Three reasoning settings—low, high, max—were tested with the same budget.

Including zero‑score unfinished tasks: low (0.476) > max (0.460) .

Considering only completed tasks: max (0.547) > low (0.527) .

The reversal occurs because max consumes more time and tokens, leading to more timeouts, execution failures, and incomplete submissions . Thus, evaluating reasoning settings requires both quality of completed reconstructions and completion rate under budget.

Harness Comparison (DeepSeek‑V4.1‑Flash, high reasoning)

Using Codex vs. Claude Code yielded overall scores of 0.479 vs. 0.450 and event scores of 0.550 vs. 0.446 . Analysis of execution logs showed that Codex’s system prompt explicitly encourages proactive testing (“proactively run tests, lint”) and reminds the agent to reserve the final Blender launch for verification. In a rolling‑block task, Codex detected a stuck object, adjusted contact geometry and slope, ran the full program on the 20th launch, and checked output frames. In a pouring task, it iteratively adjusted the inflow nozzle and verified the final frame. Claude Code performed early tests but, after rewriting programs, lacked remaining launches to complete a successful verification run. This highlights that re‑running and checking after modifications is critical and must be budgeted for .

Conclusion and Future Work

PhysCodeGym shifts video reconstruction evaluation from pixel‑level similarity to executable physical scene construction with mechanistic verification . The benchmark reveals that even state‑of‑the‑art code agents produce visually plausible but physically invalid simulations, and that counterfactual testing is essential to uncover hidden errors. Performance depends not only on model capability but also on inference settings and harness‑driven execution strategies.

The project is open‑sourced: code, all 156 standardized reference videos, task definitions, physical criteria, the Harbor task‑building tool, and evaluation implementation are available on GitHub ; the dataset can be downloaded from Hugging Face . Future extensions will support more simulation engines and richer physical interactions.