Optimizing Real-Time Ad Bidding with Reinforcement Learning: A Deep Dive

Introduction

In computational advertising, real‑time bidding (RTB) is a crucial transaction mechanism where advertisers submit bids for ad slots; the highest bidder wins but pays the second‑highest price. The dominant pricing model is cost‑per‑click (CPC).

Advertisers must design a bidding strategy that respects a budget constraint—total spend cannot exceed a fixed amount—while maximizing total clicks. Finding the optimal strategy under this budget is a valuable problem.

Problem

Model the advertiser’s bidding‑strategy optimization using reinforcement learning and implement an example with a deep neural network.

Analysis and Solution

The following figures illustrate the reinforcement‑learning model, the value function, and the network architecture used to solve the problem.

The value function satisfies the Bellman equation shown in the fourth figure, guiding the optimal bidding policy under the budget constraint.

References

CAI H, REN K, ZHANG W, et al. Real‑time bidding by reinforcement learning in display advertising. Proceedings of the 10th ACM International Conference on Web Search and Data Mining, 2017: 661‑670.

WU D, CHEN X, YANG X, et al. Budget‑constrained bidding by model‑free reinforcement learning in display advertising. Proceedings of the 27th ACM International Conference on Information and Knowledge Management, 2018: 1443‑1451.