GPT-6 Astra as Robot Brain: 95% Pick-Place Success but 10% on Precision Tasks

GPT-6 Astra as a General-Purpose Robot Brain

Recent experiments show that OpenAI's GPT-6 Astra can control diverse robot hardware without robot-specific training. A $150 SO-101 arm drew the Golden Gate Bridge by planning stroke order and using visual feedback. Sam Altman retweeted the demo, highlighting broad interest.

Key Experiments

Physical in-context learning (CMU, Wenli Xiao): A human demonstration video was fed to Codex; Astra then controlled a real arm to replicate the task in one shot, demonstrating "physical ICL" — transferring few-shot learning to the physical world.

Real-to-sim (Lingxiao Guo): Multi-view robot data given to Astra produced camera calibration, scene reconstruction, and a MuJoCo environment supporting contact simulation and motion replay.

Dexterous hand drawing (Dmytro Hrybov): Kinova Gen3 arm with Shadow Hand drew Picasso's dove, handling grip pose, finger-pencil friction, and tip-paper contact.

Dual-arm pick-and-place (RoboCurve): Two I2RT YAM arms, three camera views, and proprioception fed to Astra. It output 6-DoF end-effector poses (x, y, z, yaw, pitch, roll) plus gripper state; inverse kinematics converted to joint commands. Over 20 trials, Astra succeeded 19 times (95%) vs. Fable 5.1's 8 successes (40%).

Control Architecture and Latency Limits

Astra operates at the task-planning level, outputting target end-effector poses while traditional IK handles joint-level control. Yu Xiang (ex-NVIDIA) notes this deeper integration into manipulation is a promising direction. However, when tested at 50 Hz joint-level control on a Unitree Go1 quadruped, Astra's inference latency forced simulation pauses; 250 inferences yielded only ~5 seconds of walking.

The "Last Millimeter" Problem

In RoboCurve's tests, Astra achieved 95% success on coarse block-in-bowl but only 10% on precise puzzle-piece insertion (2/20). Failures occur in the final millimeters: slight pose errors prevent alignment, and contact dynamics disrupt subsequent actions. Physical Intelligence similarly finds tasks like M3 screw driving, cable insertion, and zip-tying require online reinforcement learning fine-tuning for fine contact.

Industry Context: OpenAI's Robotics History and the Foundation Model Debate

OpenAI previously developed Dactyl (2018) for in-hand manipulation and Rubik's cube solving but shut down its robotics team due to data scarcity — robot data is expensive, slow, and lacks the scale of internet text. After scaling language, code, vision, and agent capabilities, OpenAI re-enters robotics (Sam Altman confirms humanoid plans). Zeeshan Zia (Amazon Alexa Principal Scientist, Retrocausal co-founder) suggests "the OpenAI of robotics may be OpenAI itself, not Physical Intelligence or Skild." Those startups pursue robot-specific foundation models trained on massive embodied datasets; Astra demonstrates an alternative: a general model transferring its world knowledge to physical action.

Conclusion

Astra excels at perception, reasoning, and high-level planning but struggles with high-frequency feedback and fine force control. The likely near-term architecture: Astra for understanding and planning, low-level policies for fast, stable execution. A capable AI brain still awaits a matching physical body.