Agent Lightning: Decoupling Optimizers to Empower AI Agents via Reinforcement Learning

In 2025 the AI field entered a paradigm shift as AI agents—software systems driven by large models that interact with environments in real time—moved from concept to reality. A key challenge is enabling agents to continuously learn and evolve.

At the Agentic AI Summit, Dr. Yang Yuqing, chief R&D engineer at Microsoft Research Asia, presented the design and practice of Agent Lightning , an open‑source learning system for agents that has garnered over 15,000 GitHub stars.

Reinforcement Learning’s Unique Value

Reinforcement learning (RL) differs fundamentally from supervised learning: instead of static datasets, RL agents generate data through interaction with an environment, fostering self‑exploration and the emergence of new capabilities. Recent breakthroughs such as OpenAI’s o1 models (Sept 2024) and DeepSeek R1 (Spring 2025) highlight RL’s importance for large‑model reasoning.

Agent Data: Fuel for Model Evolution

Traditional pre‑training and instruction‑tuning rely on human‑curated corpora, which are nearing saturation. Agentic data —experience generated by agents during interaction—provides a new source of training signal, driving scaling and capability growth.

From Theory to Practice: Agent Lightning Architecture

Agent Lightning tackles the gap between research prototypes and production systems by introducing an Optimizer‑Agent Disaggregation architecture:

The RL infrastructure (inference engine, task allocation) runs on a dedicated compute layer.

An intermediate layer delegates and coordinates these resources to agents, decoupling compute‑intensive tasks from application‑specific logic.

This enables agents to run “anywhere” without being tied to GPU clusters, while the compute layer operates independently.

Three Core Technical Innovations

Non‑Intrusive Experience Capturing : Leveraging OpenTelemetry, Agent Lightning repurposes observability data (trajectories, rewards) for training without modifying agent code, supporting both white‑box and black‑box agents.

Programmable Experience Pipeline : A flexible data loader allows custom reshaping, reordering, and transformation of experience data to accommodate rapidly evolving RL algorithms.

Flexible Signal Passing : An emit API lets any serializable data (rewards, warnings, custom signals) flow directly to the training engine via a shared data store.

Agent‑Native Design Principles

Agent Lightning embodies four “agent‑native” traits:

No constraints on the agent’s runtime environment (private deployment, off‑cluster execution).

No preset requirements on agent architecture or orchestration (supports multi‑agent, memory‑augmented, non‑linear workflows).

Framework‑agnostic (compatible with LangChain, Microsoft Agent Framework, CrewAI, etc.).

Zero or minimal code changes required for existing agents.

MDP Abstraction for Agent‑Native Systems

Agent Lightning adopts a Markov Decision Process (MDP) abstraction that treats both model and native code (Python, TypeScript) as part of the environment, enabling richer diversity, semantic information, and training‑inference consistency.

Learning‑from‑Experience Vision

The system extends beyond RL to a broader “Learning from Experience” loop: agents generate experience, which is processed by algorithms (RL, reflection‑based, memory‑based) to produce updated model parameters, new skills, memories, or prompts. These artifacts are persisted and injected back into agents, driving continual self‑evolution.

Practical Cases and Benchmarks

Agent Lightning demonstrates near‑zero code integration across eight diverse agent types (room‑booking, capital‑query, math reasoning, text‑to‑SQL, multi‑hop QA, etc.) and optimization methods (APO, SFT, GRPO, Tinker). Experiments show:

In‑distribution : Superior performance to native GRPO on familiar tasks.

Out‑of‑distribution : Strong results without weight updates, leveraging learned prompting and meta‑learning.

A “Training with Memory” experiment jointly updates policy parameters and a non‑parametric memory module, enabling agents to generate and reuse tips across tasks, achieving notable gains on both seen and unseen environments.

APO Skill Learning

Advanced Prompt Optimization (APO) combined with skill learning transforms a generic programming agent into a domain‑specific verifier, markedly improving success rates by automatically identifying knowledge gaps and filling them from a knowledge base.

Achievements and Outlook

Since its open‑source release, Agent Lightning has earned 15,300+ GitHub stars, topped the GitHub Trending list (Jan 20 2026), integrated with Microsoft Agent Framework, and validated training on >100 GPU clusters. The team envisions every agent thriving in a learning‑centric era, inviting community contributions.