How Deep Reinforcement Learning Powers Video Game AI: From Q‑Learning to Atari Mastery

Deep Reinforcement Learning in Video Games

Scenario Description

Games are one of the most representative and suitable application domains for reinforcement learning (RL), encompassing all RL elements: environment (game state), actions (player inputs), agent (program), and feedback (score, win/loss). Playing video games directly from raw pixels is a hallmark of AI maturity. Atari, a popular 1970s‑80s console, provides a simple visual environment and a mature emulator, making it ideal for testing RL algorithms. In a discrete‑time setting, each step yields a screen frame, an action command (e.g., up, down, fire), and a reward. The huge state space of raw pixels prevents traditional methods, prompting DeepMind’s 2013 introduction of deep reinforcement learning.

Problem Description

What is deep reinforcement learning, how does it differ from traditional RL, and how can it be used to play video games?

Answer and Analysis

Traditional RL mainly uses Q‑learning. Deep reinforcement learning (Deep Q‑Learning) retains the Q‑learning framework but replaces the tabular Q‑function with a deep neural network and adds techniques such as experience replay to accelerate convergence and improve generalization.

Classic Q‑learning diagram:

To compare with Deep Q‑learning, the final step can be expressed equivalently:

Deep Q‑learning (red parts highlight differences from traditional Q‑learning):

Differences lie mainly in the details of sub‑steps. Traditional Q‑learning obtains the current state directly from the environment observation, whereas Deep Q‑learning first processes the observation (e.g., stacking frames, resizing, normalizing) to produce the input for the Q‑network. This preprocessing is essential when using raw Atari frames.

Conclusion

This section introduced three methods in reinforcement learning, noting that many other approaches share the same underlying ideas.

Reference:

Antonoglou, I., Graves, A., Kavukcuoglu, K., Mnih, V., Riedmiller, M.A., Silver, D., & Wierstra, D. (2013). Playing Atari with Deep Reinforcement Learning. CoRR, abs/1312.5602.