Avoiding Deceptive Conclusions in LinkedIn Advertising AB Tests and the Budget‑Splitting Method

Background

LinkedIn, the world’s largest professional network with nearly 900 million users and 26 million companies, runs a massive volume of advertising experiments across UI changes, AI model updates, backend tweaks, and bug fixes. Over 30 000 metrics are tracked daily and thousands of experiments run simultaneously.

Ads can be served on LinkedIn’s own platform or on partner sites (LinkedIn Audience Network). Advertisers target specific industries or professional audiences, and the platform’s marketing solutions aim to deliver precise brand exposure.

Experiment‑Related Challenges

LinkedIn’s strong experimentation culture (“Data is in our DNA”) leads to many concurrent tests. However, large‑scale ad experiments face issues such as cannibalization, where an uplift observed in a small‑scale test diminishes when traffic is scaled to 100 %.

Example: a 0.5 % lift in revenue at 5 % traffic may shrink dramatically at full rollout due to cannibalization. The effect can cause the observed lift to be up to 2.3 × the true effect.

Detecting cannibalization requires gradual traffic ramp‑up (1‑5 % → 25‑50 % → 100 %). If the lift shrinks as traffic increases, cannibalization is likely.

Industry Solutions

Unbiased data : Use raw data without extra processing.

High statistical power : Design experiments that can reliably detect differences.

Stability and reproducibility : Ensure experiments can be repeated with consistent results.

Solution 1 – Model‑based correction: Use a predictive model to extrapolate 100 % results from early ramp data, but model error can be large.

Solution 2 – Campaign‑level randomisation: Randomly assign whole campaigns to treatment or control, which reduces sample size and may still be biased.

Solution 3 – Daily alternating experiment: Switch all users to control one day, treatment the next. This yields few data points and can introduce bias due to daily budget differences.

LinkedIn’s Innovation – Budget‑Splitting Method

The budget‑splitting experiment halves both the advertiser’s budget and the user pool, creating two independent markets (treatment and control). Users in each market only see ads from their respective side.

Assumptions:

The two markets do not interfere with each other (approximate independence).

Halving budget and audience yields proportional response, allowing treatment metrics to be scaled to the full market.

Under these assumptions, the measurement is unbiased and statistical power is very high—often 30 × higher than daily‑alternating experiments.

Practical results show extremely low variance, enabling detection of revenue differences as small as a few cents per user. Compared with traditional randomised user experiments, budget‑splitting removes cannibalization bias and uncovers much smaller effect sizes.

Limitations: the method assumes market independence and proportional response; over‑splitting the market can violate these assumptions. Typically up to four independent budget buckets are used.

Q&A Highlights

Q1: Should actual spend differences be corrected when budgets are split?

A: Budget utilisation may drop when the split is too fine; practical implementations limit splits to four buckets.

Q2: How many buckets are feasible and what sample size is required?

A: Too few users per bucket harms power; the number of buckets is chosen based on power calculations and traffic availability.

Q3: Does splitting budget affect the lift curve?

A: For large spend levels the lift curve remains stable; the system smooths spending to keep the curve flat.

Q4: How to handle ad cold‑start in AB tests?

A: If early‑day effects are strong, consider discarding the first days or performing longitudinal studies to isolate stable effects.

Q5: How to convince business stakeholders of a new experiment design?

A: Quantify expected cost‑benefit, present clear statistical gains, and show how the design reduces risk and improves decision‑making.

In summary, LinkedIn’s budget‑splitting experiment is a preferred solution for ad‑product testing, offering unbiased results, high statistical power, and applicability beyond advertising to other markets such as recruitment.