Unlocking Reinforcement Learning: Core Concepts, Algorithms, and Real‑World Applications

1. What Is Reinforcement Learning

In an unknown environment there is an agent that interacts with the environment to receive rewards . The agent’s goal is to maximize cumulative rewards by taking appropriate actions . Reinforcement learning aims to learn an optimal policy through experimentation and feedback .

The objective is to discover a policy that maximizes future rewards by learning from trial‑and‑error.

Figure 1. Agent interacts with the environment to maximize cumulative reward.

1.1 Key Concepts

We formally define several basic concepts.

Agent takes actions in an environment . The environment’s response is defined by a model (known or unknown). After each action the environment provides a reward as feedback.

The model defines the reward function and transition probabilities. When the model is known we have model‑based RL ; otherwise we have model‑free RL .

Policy guides the agent to select actions that maximize total reward. Each state has a value function estimating the expected return from that state.

Figure 2. Summary of RL methods: which parts (value function, policy, environment) are modeled.

Interaction generates a trajectory of states , actions , and rewards . The sequence is called an episode (or trial/trajectory) and terminates at a terminal state.

1.2 Markov Decision Process (MDP)

Almost all RL problems can be described as an MDP, where the future depends only on the current state (Markov property). An MDP consists of five elements: a set of states, a set of actions, a transition‑probability function, a reward function, and a discount factor.

Figure 3. Agent‑environment interaction in an MDP.

1.3 Bellman Equations

Bellman equations decompose a value function into immediate reward plus discounted future reward.

1.3.1 Bellman Expectation Equation

The recursive update splits the state‑value and action‑value functions, allowing policy‑based extensions.

Figure 5. How the Bellman expectation equation updates state and action values.

1.3.2 Bellman Optimality Equation

When only the optimal value is of interest, the equation selects the maximum over possible actions.

2. Common Approaches

2.1 Dynamic Programming

If the model is fully known, dynamic programming iteratively computes value functions via Bellman equations and improves the policy.

2.1.1 Policy Evaluation

Computes the state‑value function for a given policy.

2.1.2 Policy Improvement

Uses the value function to greedily improve the policy.

2.1.3 Policy Iteration (Generalized Policy Iteration)

Alternates policy evaluation and improvement until convergence.

2.2 Monte‑Carlo Methods

Monte‑Carlo learns from complete episodes without modeling the environment, estimating returns by averaging observed returns.

2.3 Temporal‑Difference (TD) Learning

TD learning is model‑free and updates from incomplete episodes using bootstrapping.

2.3.1 Bootstrapping

TD updates target values based on existing estimates rather than full returns.

2.3.2 Value Estimation

TD target updates the value function with a learning‑rate‑controlled step.

2.3.3 SARSA (On‑Policy TD Control)

Updates Q‑values using the current policy’s actions.

2.3.4 Q‑Learning (Off‑Policy TD Control)

Updates Q‑values using the maximal estimated action value, independent of the behavior policy.

Figure 6. Backup diagrams for Q‑learning and SARSA.

2.3.5 Deep Q‑Network (DQN)

DQN stabilizes Q‑learning with experience replay and periodic target‑network updates.

Figure 7. DQN with experience replay and target network freezing.

2.4 Combining TD and MC Learning

Multi‑step TD methods use several future steps to estimate returns, weighting them with a discount factor.

2.5 Policy Gradient

Policy‑gradient methods directly learn the policy parameters by maximizing expected return.

2.5.1 Policy Gradient Theorem

Provides the theoretical foundation for gradient‑based policy optimization.

2.5.2 REINFORCE

Monte‑Carlo policy gradient that updates parameters using sampled returns, often with a baseline to reduce variance.

2.5.3 Actor‑Critic

Combines a critic that learns a value function with an actor that updates the policy.

2.5.4 A3C (Asynchronous Advantage Actor‑Critic)

Parallel training of multiple actors with a shared global network; uses advantage estimates as baselines.

2.6 Evolution Strategies (ES)

ES optimizes policy parameters without gradient back‑propagation, relying on random perturbations and fitness evaluation.

3. Known Problems

3.1 Exploration‑Exploitation Dilemma

Balancing exploration and exploitation is crucial; common solutions include ε‑greedy and parameter perturbations.

3.2 Deadly Triad Issue

Combining off‑policy learning, function approximation, and bootstrapping can cause instability; techniques like experience replay and target networks help mitigate this.

4. Case Study: AlphaGo Zero

AlphaGo Zero uses a deep residual network and Monte‑Carlo Tree Search, learning solely from self‑play without human data.

Training minimizes a loss that combines policy and value errors, leading to superior performance over the original AlphaGo.