Advances in Automated Bidding and Auction Mechanisms for Online Advertising

Online advertising involves multi‑party optimization (media, platform, advertisers). Automated bidding and intelligent auction mechanisms are essential for maximizing marketing goals while maintaining ecosystem health.

Early solutions treated budgeting as a control problem (PID‑based). Subsequent approaches formulated the problem as a constrained convex optimization (LP) and solved it online, but required accurate traffic forecasts.

Reinforcement learning (RL) emerged as a natural fit for the sequential decision nature of bidding. Initial RL methods used model‑based or model‑free techniques on request‑level data, facing challenges such as massive state space, sparse delayed rewards, and high training cost.

To overcome the impracticality of full‑online training, a Simulation RL‑based Bidding (SRLB) pipeline was introduced, using a simulated auction environment to generate training data.

Recognizing the distribution gap between simulation and real traffic, an Offline RL‑based Bidding paradigm was proposed. It directly learns from logged online decisions, applying Conservative Q‑Learning (CQL) and Implicit Q‑Learning (IQL) with data‑support weighting to mitigate over‑estimation and out‑of‑distribution issues.

Further reduction of interaction cost led to Sustainable Online RL (SORL), which safely explores in the live environment using Lipschitz‑smooth Q‑functions and a V‑CQL offline update to stabilize training.

With the rise of large generative models (e.g., ChatGPT), a Generative Bidding approach (AIGB) was developed. It treats bidding, objectives, and constraints as a joint probability distribution, training a conditional generative model on historical trajectories to produce coherent bidding policies without step‑by‑step error accumulation.

The evolution can be summarized in four generations: classic control, planning/LP, RL‑based, and generative‑model‑based, each improving decision strength and adaptability to complex, multi‑slot, and multi‑objective auction settings.

Additional research directions include multi‑agent coordination, fairness across advertisers, bid shading for first‑price auctions, and two‑stage auction designs that balance efficiency and latency.

The article concludes that continuous innovation—from control theory to generative AI—drives the future of intelligent ad decision systems.