AdPilot: Fully Autonomous Advertising Delivery via Agentic Reinforcement Learning (KDD 2026)

Background and Motivation

Online advertising is a core revenue source for internet platforms and a key driver of enterprise growth. In practice, ad delivery relies heavily on manual configuration of bids, budgets, targeting, and creatives, leading to decision latency, budget waste, and unstable delivery. Existing automation tools address isolated sub‑tasks such as bidding or budget allocation but cannot achieve end‑to‑end optimization.

Task Definition: Modeling Advertising as a Sequential Decision Problem

AdPilot formalizes ad delivery as a finite‑horizon Markov Decision Process (MDP) defined by a four‑tuple:

State space: all information required for decision making, including product description, category, and historical memory (past states, actions, and effects).

Action space: concrete ad operations such as creating, pausing, deleting ads, and adjusting bids, budgets, or placements.

Transition function: models the evolution of ad states under stochastic market dynamics.

Reward function: measures business benefit (e.g., conversion gain, cost efficiency, ROI improvement) of an action in a given state.

Advertising feedback (conversions, revenue) is delayed, so the observation function maps actions and environment randomness to delayed outcomes. The learning objective is to find a policy that maximizes the expected cumulative reward over trajectories.

Technical Solution

Overall Framework

AdPilot consists of three core modules that jointly enable end‑to‑end optimization:

Memory & Reflection: maintains short‑term operational state and long‑term experience across cycles.

Reasoning Enhancement: injects advertising domain knowledge to generate interpretable reasoning.

Reinforcement Learning Engine: uses a hybrid reward mechanism to directly optimize business objectives.

When a delivery task arrives, the system first extracts the short‑term state from the memory store, reflects on long‑term experience, and then, given the current state, business goal, and aggregated context, outputs the final ad action.

Memory and Reflection

AdPilot employs a two‑layer memory mechanism:

Short‑term memory: converts raw historical data into a structured table. By reusing column headers and appending rows, token length is reduced from ~20k to ~12k (≈40% savings) while aligning metric columns across timestamps to improve temporal reasoning.

Long‑term memory: at the end of each day, an LLM‑driven summarizer merges the day’s table with the previous day’s summary within a fixed token budget, eliminating redundancy, preserving cross‑day trends, action‑effect patterns, and systematic anomalies, thereby supporting effectively unlimited horizon memory with minimal overhead.

Building on the Reflexion paradigm, AdPilot introduces a Reflector‑Actor collaboration. The Reflector runs hourly, analyzes internal metric trends, identifies inefficient actions or anomalies (e.g., sudden exposure drop, ROI deviation), and generates diagnostic reflections. The Actor incorporates the original context and the reflection to produce the final ad action. Both share the same base model, differentiated only by role prompts.

Reasoning Enhancement

Standard instruction‑tuning data lack advertising‑specific details, so AdPilot constructs a large domain‑specific reasoning dataset. Real ad system logs are sampled, paired with advertiser goals (e.g., target ROI), and enriched with expert examples. The flagship model Qwen3‑235B‑A22B‑Thinking generates few‑shot reasoning trajectories, each filtered through multiple generations.

Samples are scored using a “Rubrics as Rewards” scheme that evaluates key reasoning point recall and precision, token efficiency, output format compliance, and environment feedback, with rejection sampling to ensure high‑quality data for the subsequent RL stage.

Reinforcement Learning

The training pipeline follows a two‑stage approach: supervised fine‑tuning (SFT) for cold‑start, then Group Relative Policy Optimization (GRPO) for RL. GRPO samples a set of candidate actions from the current policy, normalizes rewards across the group, and updates the policy toward higher‑reward actions.

To address delayed business feedback and misaligned training signals, a dedicated advertising environment simulator is trained on 500 k historical trajectories using Qwen2.5‑32B. The simulator achieves ~95% feedback accuracy on a held‑out test set, providing timely and reliable simulated signals for policy optimization.

The reward function comprises three components:

Format reward: enforces strict JSON schema compliance (binary 0/1).

Reasoning reward: computes F1 over task‑specific checkpoints and penalizes token bloat.

Result reward: combines short‑term business signals (conversion lift, cost efficiency, ROI improvement) with long‑term goals (overall plan fulfillment) while penalizing abrupt parameter changes to ensure delivery stability.

These components are weighted to produce the final reward.

AdBench Benchmark

To fill the gap of a standardized end‑to‑end ad‑delivery evaluation, the team releases AdBench, the first benchmark for advertising agents that jointly assesses reasoning and business capability. AdBench covers three capability dimensions—cold‑start, cost control, steady‑state maintenance—spanning 38 business sub‑scenes and 7,600 test instances.

Evaluation dimensions include instruction compliance, numeric perception, hallucination detection, reasoning correctness, and action consistency. Data sources consist of 60% real production logs, 20% manually crafted edge cases, and 20% historical failure cases, annotated by senior advertisers with tolerance ranges for correct actions.

Overall score = weighted sum of business score (average Pass@k across scenes) and reasoning score (average of five LLM‑as‑Judge dimensions, with DeepSeek‑V3.2 achieving 93% human verification accuracy).

Experimental Results

Offline experiments on AdBench show that AdPilot achieves comprehensive superiority:

Overall score improves by 4.30%–11.76% over the strongest baseline.

Despite using a 32B‑parameter backbone, AdPilot’s Pass@1 score exceeds a 235B‑parameter Qwen3 model by 6.5% and surpasses the closed‑source Gemini‑3‑Pro by 4.3%.

General LLMs (e.g., Gemini‑3‑Pro) attain high reasoning scores (77.30%) but low business scores (50.20%), highlighting a gap between generic reasoning and domain‑specific decision making. AdPilot attains 79.63% reasoning and 56.90% business scores, confirming that a domain‑aligned small model can match or exceed large generic models.

Ablation studies validate the complementary role of each module:

Removing Memory & Reflection drops business score by 8.89%.

Removing Reasoning Enhancement reduces reasoning score by 16.32% (the largest decline).

Removing RL training reduces the overall score by 11.76%, underscoring its importance for long‑term objective alignment.

Conclusion and Outlook

AdPilot is the first fully autonomous end‑to‑end advertising agent, achieving a paradigm shift from human‑in‑the‑loop to human‑on‑the‑loop through structured memory, LLM‑driven reasoning, and a GRPO‑based RL engine. The team will open‑source the AdBench benchmark to provide a standardized evaluation protocol for future research in autonomous ad delivery.