Embodied Agent Self-Evolution: Feedback Granularity, Skill Libraries & Multi-Candidate Search

Embodied agents operating in real physical environments face a fundamental dilemma: the infinite diversity of layouts, object states, lighting, and execution noise makes it impossible for any pre-defined skill or policy to cover all situations. Therefore, agents must possess the ability to continuously learn from their own execution trajectories — i.e., skill self-evolution.

Introduction

Two recent works approach this problem from different angles. EmbodiSkill (Tsinghua AIR) focuses on skill-aware reflection and targeted revision. ASPIRE (NVIDIA GEAR Lab) focuses on skill discovery and accumulation within a code-as-policy framework. Together they reveal three key design judgments for continuous evolution: feedback granularity determines evolution quality, skill libraries are becoming the core carrier of knowledge accumulation, and multi-candidate exploration is the critical path to robust evolution. These three judgments form an organic system: fine-grained feedback drives skill library deposition; multi-candidate competition selects winners for the library; the library in turn feeds back to improve feedback granularity.

Feedback Granularity Determines Evolution Quality

In skill self-evolution systems, the core design decision is how to convert an execution trajectory into an update signal. Traditional methods package the entire trajectory with a success/failure label and ask a large model to generate a holistic skill revision. This seems direct but carries serious risks in physical worlds.

The root cause is that task failure reasons are highly entangled. A grasp failure may stem from a missing precondition check in the skill description (skill content issue), perception jitter causing localization offset, or the motion planner encountering an unreachable configuration (execution process issue). If the system cannot distinguish these, it may misjudge an occasional execution lapse as a skill defect and erroneously "correct" an originally valid skill.

EmbodiSkill solves this attribution problem by classifying trajectory reflection into four categories: Discovery (new content), Optimization (improve existing content), Skill Defect (skill itself flawed), and Execution Lapse (skill correct but execution failed). Different reflection types trigger completely different update operations: the first three modify the skill body, while Execution Lapse only updates a skill appendix — an auxiliary structure reminding the executor of specific cautions, never touching core rules. On the ALFWorld benchmark, this fine-grained attribution lifted task success rate from 78.36% (coarse-grained update) to 93.28%, a 19% relative improvement.

ASPIRE addresses feedback granularity at the execution engine layer . It designs a fine-grained multimodal execution trace recorder — every perception, planning, grasp, and motion control call logs inputs, outputs, return values, and key frames. This lets the agent drill down layer by layer: from "task failed" to "planning failed" to "target point lies in collision buffer". In a "navigate and pick up radio" debugging case, the trace showed perception succeeded but navigate_to_pose repeatedly returned planning errors; further inspection revealed the navigation goal fell inside the table edge's collision buffer (~20 cm). Based on this diagnosis, the agent wrote a multi-angle approach strategy instead of blindly modifying perception or grasp modules.

The shared insight: evolution quality fundamentally depends on the resolution of feedback along the causal dimension. If feedback is just success/failure plus raw trajectory text, the evolution model must guess the problem from noise; if feedback is already structured by content error, perception failure, planning infeasibility, etc., repair accuracy and evolution stability improve significantly.

Practical implication: In our embodied agent framework's reflection module review, we recommend three concrete "attribution dimension" checks:

Task-level attribution: Does the reflection output locate "which sub-task" (e.g., "grasp cup failed" vs "entire cooking failed")?

Failure-mode attribution: Does it distinguish "planning error / execution error / perception error / external disturbance"?

Actionable-advice attribution: Does it specify "what exact code/parameter/skill to modify next time" (rather than vague "try again")?

Skill Library as Knowledge Organization

In traditional designs, the skill library is often treated as an auxiliary component of the execution model — storing predefined policies for retrieval at inference. Both works show that when the library gains the full capability of "receive evidence → attribute classification → selective update → cross-task reuse", it becomes the core carrier of knowledge accumulation.

EmbodiSkill splits skills into body and appendix, turning updates from wholesale overwrites into targeted edits. Verified core rules settle in the body, untouched by execution-lapse feedback; easily overlooked execution reminders go into the appendix. This separation guarantees skill stability — correct rules are not polluted by occasional failures.

ASPIRE's skill library exhibits stronger emergence. It does not predefine a skill taxonomy; the agent discovers reusable repair patterns during debugging. The paper shows skills spanning localization disambiguation, motion primitive construction, navigation recovery, scene understanding, debugging workflows — categories that grew naturally from task accumulation. Crucially, experiments show a direct positive correlation between library size and zero-shot transfer performance: as the library expanded from 0 to 90 task sources, success rate rose monotonically from 4.7% to 30.5%.

This points to a deeper shift: the skill library is evolving from a container for auxiliary information into the agent's long-term knowledge body. When the execution model stays fixed (parameters frozen) while the skill library continuously accumulates, overall system capability is no longer bounded by a single model's capacity but by the richness of the knowledge base. This "fixed reasoning engine + continuously evolving knowledge body" architecture has far greater long-term potential than relying solely on model parameter scaling.

Our actionable steps: Reconstruct from "policy retrieval component" to "knowledge accumulation body" in three steps:

Step 1: Separate existing skills into "body (core executable code) + appendix (interface specs / calling examples / adaptation scenarios)" — if currently mixed, first extract appendix fields into independent JSON/YAML.

Step 2: In the reflection module, compute a "skill value index" matrix from "skill call frequency × cross-task reuse rate" and generate a report.

Step 3: Outside the frozen reasoning engine, plot "skill library size ↔ cross-task success rate" curve to quantitatively assess accumulation effectiveness.

Multi-Candidate Exploration: Beyond Single-Trajectory Reflection

EmbodiSkill's evolution spiral forms a complete loop, but each iteration is essentially a serial "one trajectory → one reflection → one update" mode. When a single trajectory's evidence carries happenstance, update quality is limited.

ASPIRE's evolutionary search offers a different approach: each round generates K candidate programs, all executed on a debugging seed set, and the best performer becomes the next round's seed. In a "place bowl on plate" task, search started at 62% and reached 86% after multiple competitive rounds. Eliminated candidates are not wasted — their failure modes are recorded to avoid repeating the same error directions.

Comparing the two mechanisms yields a design principle: skill evolution should not over-rely on a single trajectory but should validate and select among multiple candidate hypotheses. Single-trajectory reflection resembles "revising an operations manual based on one fault diagnosis"; multi-candidate exploration resembles "proposing multiple fixes, testing them in practice, selecting the winner, and standardizing the success pattern". The latter is more robust against physical environment uncertainty. The cost is increased compute, but once a skill enters the library, all subsequent tasks reuse it — this "upfront investment, amortized later" structure is reasonable at scale.

Pilot recommendations: Test multi-candidate mechanisms on two "low-cost high-value" task paths:

Pilot 1: Critical sub-tasks with >30% failure rate (e.g., "unseen object grasp", "long-horizon navigation") — generate 2-3 candidate revisions in the reflection module, run 3-5 seed validations per candidate, admit the winner to the skill library.

Pilot 2: High-frequency reusable tasks (e.g., "standard cooking procedure", "fixed patrol route") — generate 2 candidate execution paths per round in the execution module, admit the winner to the execution cache.

Evaluation metrics: Compare pre/post "same-task repeat failure rate" + "skill library growth health (no duplicates / no low-quality)", monthly review.

Open Problems

The three directions above outline the core design space for continuous evolution but leave open questions.

Skill Library Scale Management

ASPIRE shows performance climbing from 0 to 90 tasks, but at larger scales retrieval efficiency, entry conflicts, and obsolescence will become prominent. Designing adaptive validation-eviction mechanisms is key for production engineering. Consider introducing LRU (Least Recently Used) eviction based on call frequency plus periodic conflict detection as a lightweight maintenance mechanism; and add a "value index" assessment before library admission, combining EmbodiSkill's appendix-style stability constraints with ASPIRE's emergent categorization.

Verification Protocol Design

EmbodiSkill triggers updates at a fixed threshold; ASPIRE validates on a debugging seed set before admission. However, seed set selection may introduce overfitting risk. Designing representative validation sets under limited debugging resources needs further research. A hybrid approach could introduce a difficulty-stratified validation set structure beyond the seed set, and fuse the two paradigms: fast threshold triggering for low-risk skills, multi-seed validation for high-risk skills, reducing overfitting while controlling debug cost.

Zero-Shot Skill Composition

Current work follows a "extract skills from trajectories" pattern. When the library becomes sufficiently rich, could agents recombine existing skills to handle novel tasks without trajectory-driven updates? If feasible, evolution efficiency would leap — composition cost is far lower than debugging from scratch. Consider letting the agent prioritize retrieving similar library entries for combination attempts on new tasks, then feeding combination failure modes back to the reflection module as new learning signals, making "combinatorial exploration" and "trajectory-driven update" complementary rather than substitutive.

Conclusion

Continuous evolution of embodied agents cannot be solved merely by larger models or more pre-training data. It requires a system architecture that lets agents diagnose problems, distill patterns, deposit knowledge, and continuously improve from their own execution experience.

EmbodiSkill and ASPIRE converge on this goal from different paths: the former ensures evolution stability through reflection categorization and targeted updates; the latter boosts discovery efficiency via fine-grained execution traces and multi-candidate competition. Together they reveal the core design principles — feedback must be sufficiently fine, knowledge must be auditable, exploration must have competition . Pursuing these directions will enable embodied agents to truly achieve a "smarter with use" evolution loop in real physical environments.

References: [1] Ju R F, Wang X R, Ding X, et al. EmbodiSkill: Skill-Aware Reflection for Self-Evolving Embodied Agents[EB/OL]. arXiv:2605.10332, 2026-05-11. https://arxiv.org/pdf/2605.10332. [2] Lu R Y, Wu Y B, Kou E, et al. ASPIRE: Agentic Skills Discovery for Robotics[EB/OL]. arXiv:2607.00272, 2026-06-30. https://research.nvidia.com/labs/gear/aspire/assets/Aspire.pdf?v=20260630d.