Understanding the Mathematical Foundations of Reinforcement Learning

Book Structure and Logical Flow

The textbook is divided into two parts: Foundational Tools and Algorithm Implementation . It builds a logical chain that starts with basic RL concepts—states, actions, rewards, returns, and policies—illustrated by a grid‑world where a robot searches for a preset goal. These concepts are formalized within the Markov Decision Process (MDP) framework, followed by the Bellman equation for policy evaluation and the Bellman optimality equation for deriving optimal policies.

Part 1: Foundational Tools

Chapter 1 defines the basic elements (states, actions, rewards, returns, policies) and uses the grid‑world example to motivate them before presenting the MDP formalism.

Chapter 2 introduces the core concept of state value —the expected return when following a given policy from a state—and the Bellman equation as the fundamental tool for policy evaluation. Action values are also defined.

Chapter 3 presents the core concept of an optimal policy , whose state values are maximal, and the Bellman optimality equation as the direct means to obtain that policy.

Part 2: Algorithm Implementation

Chapter 4 covers three closely related dynamic‑programming algorithms that require a model of the environment:

Value Iteration —solves the Bellman optimality equation directly.

Policy Iteration —extends value iteration and serves as the basis for later Monte Carlo methods.

Truncated Policy Iteration —provides a unified framework in which value iteration and policy iteration appear as special cases.

All three share the structure of alternating value updates and policy updates , a pattern known as generalized policy iteration (GPI) .

Chapter 5 introduces the first model‑free algorithms based on Monte Carlo (MC) estimation . The simplest MC Basic algorithm can be derived from the policy‑iteration update in Chapter 4. Two more advanced MC variants are presented, and the fundamental exploration–exploitation trade‑off is discussed.

Chapter 6 fills the gap between non‑incremental MC methods and the incremental algorithms of Chapter 7 by presenting stochastic approximation theory. It covers the classic Robbins‑Monro algorithm and stochastic gradient descent (SGD) , both special cases of stochastic approximation, providing the mathematical foundation for subsequent TD methods.

Chapter 7 introduces temporal‑difference (TD) algorithms as incremental, online methods that update value estimates after each experience sample. It details Sarsa (on‑policy) and Q‑learning (off‑policy), emphasizing the distinction between on‑policy and off‑policy learning and the advantage of online updates.

Chapter 8 moves to value‑function approximation for large state or action spaces. The chapter outlines a three‑step gradient‑based optimization process: (1) define a target function for the optimal policy, (2) derive the gradient of that target, and (3) apply a gradient‑based algorithm to solve the optimization problem. Artificial neural networks are introduced as function approximators, and the deep Q‑learning algorithm is presented as a concrete example.

Chapter 9 covers policy‑gradient methods , a policy‑based approach that optimizes a scalar performance measure via gradient ascent. Compared with earlier value‑based techniques, policy gradients offer better scalability, stronger generalization, and higher sample efficiency when combined with function approximation.

Chapter 10 describes Actor‑Critic methods, which integrate policy‑based and value‑based ideas. Actor‑Critic can be viewed as an extension of the policy‑gradient algorithms from Chapter 9, retaining the same function‑approximation foundation.

Inter‑Chapter Dependencies

The chapters are tightly coupled: understanding Bellman equations (Chapters 2‑3) is prerequisite for DP algorithms (Chapter 4); DP algorithms underpin Monte Carlo methods (Chapter 5); stochastic approximation (Chapter 6) provides the theory for TD learning (Chapter 7); TD learning motivates value‑function approximation (Chapter 8); and function approximation is essential for both policy‑gradient (Chapter 9) and Actor‑Critic (Chapter 10) methods.

Repository

GitHub repository for the book: https://github.com/MathFoundationRL/Book-Mathematical-Foundation-of-Reinforcement-Learning

Code example

来源：专知
本文
约3000字
，建议阅读
5
分钟
全书分为两大部分：
基础工具
与
算法实现
。