What an 8.8 M‑User Experiment Shows About the True Limits of Agent‑Based Marketing

The article analyses a real‑world longitudinal case study (Jeunen et al., 2026) that tracked roughly 8.8 million long‑inactive users of a large multi‑service consumer app for 11 months. The experiment split users into an Agentic group (≈90 %) and a control BAU group (≈10 %).

During the first four months (active phase) marketers continuously supplied new content components, adjusted audiences, and refined strategies. In the subsequent seven months (passive phase) all human‑driven content injection and routine optimisation were halted, forcing the Agent to operate solely with the existing component library and business constraints.

The Agentic system used a Thompson Sampling bandit algorithm to balance exploitation of known effective strategies with exploration of new content‑time‑channel combinations. Outcome metrics were estimated with ML‑RATE variance reduction and the Delta method to assess statistical uncertainty.

Active phase results : Compared with the BAU rule‑based group, the Agentic group achieved a 65.3 % lift in notification‑click‑derived app opens, a 2.8 % increase in positive‑interaction days, and roughly 0.3 % gains in both intent‑behaviour days and conversion‑behaviour days. Gross Merchandise Value (GMV) showed proportional improvement without evidence of reduced per‑transaction value.

Passive phase results : After human optimisation stopped, the Agentic group still retained a 57 % lift in notification clicks, a 2.4 % rise in interaction days, and a modest 0.2 % increase in intent and conversion metrics. However, performance gradually declined toward the end of the experiment, indicating that continuous human input can add an extra 12 %–26 % uplift.

The authors interpret these findings as evidence that Agentic systems can maintain a baseline of personalized marketing for months without fresh human input, but they cannot replace strategic innovation. Human marketers remain essential for defining business goals, creating new content assets, and periodically recalibrating the decision space.

Databricks’ June 2026 launch of CustomerLake – an “Agentic CDP” embedded in a Lakehouse – is positioned as a product response to the experiment’s insights. The product promises to move beyond the traditional “Golden Record” (who the customer is) toward a “Golden Context” that also encodes what the business wants to achieve, past actions taken, and real‑time feedback. CustomerLake’s architecture includes Profile Agents (data cleaning, identity resolution) and Campaign Agents (audience building, action recommendation, cross‑channel execution) and introduces the notion of “Infinity Campaigns” that continuously adapt to new user signals.

Nevertheless, the article warns of several risks: erroneous identity resolution, delayed behavioural data, stale customer tags, or mismatched metric definitions can be amplified by the Agent. Attribution remains a challenge—clicks or conversions cannot be fully ascribed to a single message. Moreover, trade‑offs between click‑through, conversion, profit, unsubscribe risk, and long‑term user value must be governed by clear business guardrails, with capabilities for pause, audit, and rollback.

In summary, the 8.8 M‑user, 11‑month study confirms that Agentic marketing can sustain incremental impact after human optimisation ceases, but long‑term effectiveness ultimately depends on periodic human‑driven strategy refreshes. The emerging “Agentic CDP” model therefore redefines the division of labour: agents maintain scalable baseline decisions, while humans act as the multiplier that drives growth through strategic innovation.