How Generative Reinforcement Learning is Revolutionizing Real-Time Bidding

Real‑Time Bidding Challenges

In a real‑time bidding (RTB) advertising system, the bidding module links advertiser goals (e.g., conversion rate, ROI) to traffic matching, directly affecting ad ranking and platform traffic allocation efficiency.

The core challenges are:

Spend wisely: keep daily spend within budget while minimizing cost per conversion.

Unpredictable future: traffic and competitor behavior cannot be foreseen, requiring dynamic bid adjustments based on real‑time spend and cost data.

Sequential impact: each bid influences ad display, consumption, and remaining budget, forming a complex sequential decision problem.

Three Generations of Bidding Algorithms

First generation (PID): analogous to cruise control, adjusts bids based solely on current vs. target speed, simple but inflexible.

Second generation (MPC): like adaptive cruise control, predicts short‑term traffic conditions to adjust bids, yet still prone to local optima.

Third generation (Reinforcement Learning): comparable to an AI driver trained on massive offline data, learns optimal actions to maximize cumulative reward, offering safety and the ability to discover superior strategies.

Since 2025, Kuaishou has fully deployed generative reinforcement‑learning bidding, boosting ad revenue by over 3%.

Why Add Generative Methods?

Standard RL in bidding is “one‑dimensional”, using only the current state and ignoring rich sequential information. Generative models (e.g., Transformers, Diffusion) excel at modeling complex sequences but depend heavily on data quality and struggle to align with optimization objectives. Combining the two can leverage the strengths of each.

Generative Reinforcement‑Learning Framework

The framework introduces two directions:

Generative Model as a World Model: creates a digital sandbox to simulate outcomes of different bidding strategies, generating abundant training data.

Generative Model as Policy: directly models the bidding policy to better exploit sequential state information.

Decision Transformer (DT) treats bidding like next‑token prediction, while Diffusion models generate future state trajectories and infer the current optimal bid.

Challenges of Generative Models

Reliance on high‑quality datasets leads to out‑of‑distribution (OOD) issues.

Difficulty aligning generated trajectories with the overall optimization objective.

GAVE Algorithm (Score‑based RTG + Value‑Guided Exploration)

To address dataset quality, GAVE incorporates a Score‑based Return‑to‑Go (RTG) module that flexibly adapts to multiple advertising goals (CPA, ROI, etc.). It also adds a value‑function‑guided action exploration mechanism that estimates long‑term value for both original and exploratory actions, steering the model toward the higher‑value choice.

CBD Algorithm (Causal Auto‑Bidding via Diffusion Completer‑Aligner)

CBD tackles the alignment challenge by introducing a Completer that, given a random decision step t, completes future state trajectories using a diffusion model, and an Aligner that adjusts generated trajectories based on a learned trajectory‑level reward model to better match the target objective.

Experimental Results

Offline tests on the AuctionNet dataset show GAVE and CBD outperform baselines (DT, other generative methods) across various budget settings, with GAVE achieving the best conversion value improvements. Ablation studies confirm the necessity of both the Score‑based RTG and the exploration mechanisms.

Online A/B tests in large‑scale ad systems demonstrate that both GAVE and CBD deliver significant gains: in Cost‑cap scenarios, consumption increased by 2.0% and CPA compliance by 1.9%; in No‑bid scenarios, consumption rose by 0.8% with a 3.2% increase in expected consumption.

Future Outlook

Two major directions are envisioned: (1) a foundational bidding large model trained on multi‑scenario, multi‑objective data using DT or Diffusion architectures to exploit scale; (2) a bidding inference large model that incorporates large‑language‑model reasoning for enhanced interpretability and decision‑making capabilities.