Can General LLMs Replace Robot Brains? First AA-Robotics Benchmark Reveals Gaps

AA-Robotics v0.1 Benchmark Overview

On October 14, 2024, the LLM evaluation organization Artificial Analysis released the first AA-Robotics v0.1 benchmark, testing five frontier large language models — GPT-6 Astra, GPT-6.1 Sol, Gemini 4 Argon, Claude Opus 5.5, and Claude Fable 5.1 — on eight real-world manipulation tasks using a Hiwonder NexArm 6-DoF robot arm equipped with a gripper and four cameras (three around the workspace, one on the wrist). Each model received only the arm's name, an HTTP API for communication, and the task description; they had to perceive the scene, write and execute control code, and complete the task zero-shot, without any task-specific fine-tuning. GPT models ran in Codex, Claude used Claude Code, and Gemini used Antigravity.

Overall Scores and Task-Level Results

The composite scores (out of 100 per task, partial credit allowed) were: GPT-6 Astra 69.4 , GPT-6.1 Sol 64.6 , Gemini 4 Argon 55.3 , Claude Opus 5.5 48.8 , Claude Fable 5.1 46.3 . Notably, Astra earned perfect scores on four tasks, while Sol earned perfect scores on five tasks despite a lower total. The two easiest tasks — pressing a red button and placing a crocodile toy into a box — were solved by all five models. Beyond those, performance diverged sharply: Astra could pour liquid and write "hello world" but scored zero on measuring a box; Sol completed block stacking, shape matching, and writing but failed at pouring. No model achieved a perfect score on the block-stacking tower task; Sol once built a valid tower but continued adjusting it, causing the task to be marked as failed.

Time and Cost Disparities

Completing a single task took the models an average of ~30 minutes (Gemini 4 Argon averaged ~52 minutes), whereas a human teleoperation baseline required only ~20 seconds — nearly two orders of magnitude faster. For the simple button-press task, Astra needed 2.2 minutes; Claude Fable 5.1 spent ~10 minutes just understanding the scene and planning before its first move. The average API cost per task was $16.36 for Astra versus $1.93 for Sol , a difference of 8.5× for only a 4.8-point score gap. For robots that must operate continuously in homes or warehouses, the difference between a one-off success and affordable, repeatable daily operation is critical.

Where the Time Goes: Perception, Planning, and Motion

Artificial Analysis broke down Astra's time allocation: ~20% environment/arm understanding, ~59% action planning, and only ~13% actual arm motion. Sol showed a similar pattern: 61% planning, 13% motion. The models behave like a person entering a lab for the first time: they observe the arm, write code, probe positions, verify camera feedback, and only then issue the next command. The physical motion itself often lasts seconds, but the preceding reasoning stretches to minutes.

Grasp Reliability and Failure Modes

Across six grasping tasks, Astra's grasp success rate was 45% , Claude Fable 5.1 44% , and Sol only 28% . Even when the target was located, the gripper often closed beside the object or the object slipped after being lifted. Recorded failure clips show the arm pushing an already-placed toy out of the target zone while trying to adjust it, the wrist colliding with a tray making a bottle harder to reach, and the pour action not accounting for the shifting liquid stream, spilling the last drops outside the cup. These errors stem from the physical world's continuous dynamics — sliding, tipping, deformation, friction, and collision — which cannot be fully resolved by "thinking longer" at a high level. Real-time correction based on proprioceptive and tactile feedback is needed, not just re-planning from vision.

Industry Perspectives: Why Specialized Embodied Models Still Matter

The benchmark fuels an ongoing debate: as general LLMs gain visual, spatial, and coding abilities, do companies building dedicated Vision-Language-Action (VLA) models still have a moat? At the September 2024 Bund Summit, Wang Qian (founder of Heterogeneous Robotics) argued that model capabilities come from data. Code models improve because of massive code corpora and mature verification; the physical world equally demands dedicated data, evaluation, hardware, and infrastructure — language proficiency does not automatically transfer. He also noted that Astra is not entirely devoid of robot data. Xu Huazhe (founder of Shell Robotics) tested Astra and found it could identify objects and pick up a chopstick, but performance dropped sharply for complex contact-rich manipulation. In his view, semantic and spatial understanding have advanced impressively, but complex physical interaction remains a separate challenge .

Limitations and Future Directions

The current benchmark does not yet pit general-purpose agents against specialized VLAs under identical conditions. Eight tabletop tasks cannot represent all robotics challenges — dual-arm coordination, deformable object manipulation, and long-horizon autonomous operation are absent. Nevertheless, AA-Robotics v0.1 provides the first systematic evidence that general LLMs can execute real physical operations, while quantifying how far they remain from efficient, reliable deployment. The first leaderboard includes no Chinese models; the next round may feature Qwen, Kimi, GLM, or DeepSeek, testing whether domestic models can match or surpass the international leaders at lower cost. The competition ahead is no longer just "who can make the arm move," but who can make it move faster, grasp more precisely, and know when to stop .

Reference: https://artificialanalysis.ai/articles/aa-robotics Reference: https://x.com/ArtificialAnlys/status/2110500262628806787