Cortex: Bidirectionally Aligned Agent Framework for Long‑Horizon Manipulation

Overview

Cortex is a bidirectionally aligned embodied agent framework designed for long‑horizon robotic manipulation, particularly complex chemical operation tasks. It connects high‑level planning and low‑level execution via a unified, structured sub‑task description that specifies the target object, location, and action constraints, ensuring that high‑level commands stay within the capabilities of the low‑level executor.

Architecture: Two‑System Division

The system is split into System‑2 (high‑level) and System‑1 (low‑level). System‑2 interprets task goals, decomposes them into structured sub‑tasks, maintains a textual memory of progress, and decides whether to continue, retry, or advance. System‑1 receives each sub‑task, translates the described object and constraints into continuous robot motions, and feeds the resulting observations back to System‑2 for memory updates and completion checks.

This design directly addresses the two core challenges for robots: (1) determining whether a planned action is feasible given the robot’s skill set, and (2) recognizing when the current action has finished before moving on.

Data: Unified Skill Paradigm

To eliminate the semantic gap between the two layers, Cortex defines 32 standardized skill primitives (e.g., Pick, Place, Pour, Unscrew, Stir) and provides language templates for each. Open‑ended commands such as “pour water into the beaker” are converted into explicit instructions that include object identity, color, spatial position, quantity, and reachability.

During data construction, sub‑task annotations are enriched with these attributes, and language compression is applied to embed key object properties and counts, improving the agent’s ability to understand spatial relations, perform object counting, and recover from failures.

The team annotated over 4,000 hours of open‑source long‑duration video and trajectory data, generated ~30 hours of simulated data with automatically labeled sub‑tasks, and rewrote existing data into the unified template using vision‑language models.

Training: Emphasizing Transition Samples

Most frames in long trajectories capture ongoing actions, so uniform frame sampling would bias the model toward “continue current action.” Cortex adopts an event‑balanced sampling strategy that densely samples frames within a ±1 second window around sub‑task boundaries while retaining regular execution frames.

In ablation experiments on the Galaxea dataset, the event‑balanced model trained on 2.72 M samples (versus 3.10 M for the baseline) achieved a higher composite score, rising from 7.58 to 8.18, demonstrating that focused supervision at transition points is more effective than sheer volume of stable execution data.

Experiments: Simulation and Real‑World Performance

Cortex was evaluated on both simulated benchmarks (LIBERO‑Long, RoboTwin 2.0) and a real dual‑arm robot platform executing zero‑shot, multi‑step chemical experiments.

On LIBERO‑Long, Cortex attained a 95.5 % success rate, surpassing the π0.5 baseline’s 92.4 % by 3.1 percentage points. On RoboTwin 2.0, it achieved 86.8 % versus 82.74 % for π0.5, confirming the benefit of aligning high‑level planning with low‑level execution.

In real‑world tests on an ARX ACONE dual‑arm system, two 14‑step procedures (liquid stirring and beaker cleaning) were run 20 times each. Cortex succeeded in 65 % of stirring trials and 55 % of cleaning trials, while end‑to‑end VLA models (π0.5 and MEM) failed due to sub‑task confusion.

Key Insights

The advantage of Cortex lies not in a single module improvement but in a systematic modeling of long‑horizon tasks: structured sub‑task interfaces constrain the planner, visual feedback with textual memory tracks execution, and a closed‑loop update continuously corrects downstream decisions. This holistic approach enables robots to maintain consistent goals, states, and actions across many steps, moving from “completing a single action” toward “completing an entire task.”