Understanding AI Agents: From Reinforcement Learning to LLM-Powered Planning

Introduction

This lecture, based on Professor Li Hongyi’s popular AI Agent video, offers a detailed textbook‑style overview of AI agents, their history, and current research directions.

What Is an AI Agent?

An AI agent receives a high‑level goal from a human and autonomously decides a sequence of actions to achieve it, continuously observing the environment and updating its plan.

Reinforcement Learning Foundations

Traditional AI agents are built with reinforcement learning (RL), where a reward function encodes the goal. However, RL requires training a separate model for each task and struggles with generalization across domains.

LLMs as Agents

With the rise of large language models (LLMs), researchers now treat LLMs themselves as agents. The model receives a textual goal, generates actions as text, and can interact with external tools or environments to achieve the goal without additional training.

Memory Modules

To avoid unbounded context, agents use a memory system consisting of three modules: Read (retrieval of relevant past experiences), Write (deciding what new information to store), and Reflection (high‑level abstraction of stored memories). This architecture mirrors retrieval‑augmented generation (RAG) but stores the agent’s own experiences.

Tool Use

Agents can call external functions (search engines, calculators, APIs) by emitting a special

Tool

token, which the system interprets as a function call. The result is fed back as

Output

and incorporated into the next generation step. This enables agents to perform tasks that exceed the knowledge stored in their parameters.

Planning and Benchmarks

Effective agents must generate and adapt plans. Researchers evaluate this ability with benchmarks such as StreamBench (sequential question answering with feedback) and PlanBench (block‑stacking and a “mystery‑block” world). Results show that older models struggle, while newer LLMs (e.g., GPT‑4, Claude, o1) achieve higher success rates, especially when combined with search or solver tools.

Challenges and Future Directions

Key challenges include handling irreversible actions, real‑time interaction, and avoiding over‑thinking (excessive internal reasoning that delays execution). Future research aims to improve world‑model simulation, dynamic memory selection, and efficient tree‑search strategies that balance exploration with computational cost.

Overall, the lecture synthesizes foundational concepts, recent advances, and open problems, making it a valuable guide for students and researchers interested in AI agents.