From RL’s Early Days to Its Future: A Four‑Stage Evolution of Reinforcement Learning

Stage 1 – Early Reinforcement Learning

About a decade ago reinforcement learning (RL) lacked a formal definition and was described merely as a method for solving Markov Decision Processes (MDPs). The dominant algorithms were value‑based DQN and policy‑based PPO. Researchers split into two camps: academia pursued general‑purpose algorithms, while industry focused on concrete applications. Numerous sub‑fields (multi‑agent RL, safe RL, etc.) emerged, many without solid practical grounding.

Stage 2 – Application‑Driven RL

Graduates of the first wave entered a “big‑application” era. Papers were expected to provide:

Exact definitions of state and action spaces.

A transition function (no single‑step, game‑over decisions).

A reward structure that demonstrated trade‑offs between short‑term and long‑term returns.

Game AI satisfied these criteria and dominated early deployments, but its market share remained limited and interest waned as the novelty faded. In other domains, incomplete simulators and the sim‑to‑real gap hindered deployment, prompting researchers to explore alternatives.

Stage 3 – Problem‑Construction Focus

Practitioners recognized that the hardest part of RL deployment is constructing a realistic problem, not solving the policy. Consequently, RL was re‑defined to include both problem formulation and policy optimization. Two representative trends illustrate this shift:

Offline model‑based RL : neural networks learn dynamics and reward models from logged data; policy optimization is a secondary step.

RL from Human Feedback (RLHF) : a reward model is trained on human preferences, then used to fine‑tune a policy.

The typical pipeline became:

Problem modeling → Data collection → Policy training → Deployment

Each stage can itself be cast as an RL problem (e.g., data collection as a curiosity‑driven exploration task). Despite the broader scope, end‑to‑end deployment remains challenging because the pipeline is often long and loosely coupled.

Stage 4 – Speculative Future and Theoretical Unification

Recent discussions blur the line between supervised learning (SL) and RL. One view frames SL as optimizing a parametric loss under a fixed data distribution, whereas RL optimizes a non‑parametric loss (the reward) under a parametric distribution (the policy). Under this view, binary classification can be expressed as an RL problem:

state = input features
action = predicted label (0 or 1)
reward = 1 if action matches true label, else 0

Because the policy gradient update for this formulation reduces to the cross‑entropy loss, SL appears as a special case of RL. If this unification holds, RL could become the default paradigm for many machine‑learning tasks.

Postscript – Community Reflections

The expansion of RL’s scope has sparked debate. Some senior researchers criticize trends such as RLHF, labeling them as “scams.” Nonetheless, the broadened definition has created research opportunities and practical jobs for RL practitioners, suggesting that the community’s growth, despite controversy, has been beneficial.