Design and Optimization Practices of the Advertising Transmission Channel at LEGO Platform

The article systematically introduces the 58 commercial advertising transmission process, noting that as business expands, ad volume grows to billions of changes daily, creating complex scenarios and stability challenges that require targeted optimizations.

Background: An ad’s lifecycle includes placement by the advertiser, notification to the retrieval platform, user exposure, interaction, billing, and eventual offline removal.

In the LEGO ad platform, billions of change messages, millions of change logs, and millions of online ads demand a highly reliable transmission channel that ensures high stability, eventual data consistency, and low latency.

1. Architecture Overview: The channel consists of a full initial build (pulling all data from the placement platform) plus incremental updates (handling all subsequent changes). Incremental tasks subscribe to change messages and write logs to a MySQL flow table; a builder polls this table to generate new index data.

2. Primary‑Backup High Availability: Two isolated pipelines (primary handling live traffic, backup as standby) allow rapid switchover when the primary experiences data anomalies, achieving near‑second recovery.

3. Stateless Incremental Processing: Each builder fetches the latest ad state directly, avoiding dependence on ordered messages or logs, reducing system complexity and improving scalability.

4. Ensuring Important Messages Are Not Lost: Message consumption is unordered and idempotent, with acknowledgments, retry queues, producer‑consumer reconciliation for critical messages, and relaxed handling for non‑critical messages to boost concurrency.

5. Flow Tiering and Isolation: Changes are prioritized (e.g., ad on/off‑line) and isolated by customer or product to prevent large‑scale bursts from affecting others.

6. Reducing Channel Latency: Index data is sharded similar to database partitioning, improving write/read performance; updates are distinguished between forward (field) updates and inverted index updates, with the latter being more costly, so non‑search‑relevant changes update only forward fields.

Horizontal scaling of builders, flow tables, and incremental tasks is enabled by the stateless design.

7. Reducing Flow Volume: Optimized scheduling replaces frequent on/off‑line tasks with a schedule management system that encodes time windows in the index, eliminating unnecessary flow entries.

Rate limiting identifies low‑value ads that generate excessive change traffic and blocks them when system capacity is exceeded.

Merge operations collapse multiple identical change logs for the same ad, dramatically reducing flow volume during spikes.

Summary: The implemented solution processes over a hundred million records daily with stable performance, and will continue to evolve to support the LEGO advertising platform’s reliability needs.