Inside ByteDance’s Traffic Platform: Powering Trillions of Real‑Time Events

ByteDance Traffic Platform Overview

The platform is ByteDance’s internal unified event‑tracking (埋点) system, covering definition, collection, production, application, and governance of the entire event lifecycle. It serves over 2,000 applications, manages more than 200,000 event types, and processes daily event volumes exceeding a trillion, saving the company hundreds of millions of yuan in costs.

Embedding (埋点) Basics

Embedding describes a series of user actions within an app, such as clicks or swipes, and enables behavior analysis, personalized recommendation, and precise marketing. The data captured includes Who, When, Where, How, and What.

Data Governance Definition

Data governance manages data throughout its lifecycle to ensure security, timeliness, accuracy, availability, and usability. It addresses both existing and incremental data, establishing a stable governance chain as the foundation for reliable data handling.

Platform Components

Embedding Content : Design, development, validation, launch, usage, and deprecation of events.

Embedding Governance : Management of stored data, focusing on cost, SLA, and compliance.

Link Side : Full‑chain collection, processing, and subscription of events across iOS, Android, and other endpoints.

Link Foundation : A self‑developed real‑time computation platform that underpins ByteDance’s trillion‑plus daily event processing.

Embedding Content Solution

The core of the solution is the embedding model, which determines the quality of design, development, testing, and usage. User pain points include difficulty finding events, unclear metrics, and trust issues for consumers, and long production chains, model implementation challenges, and lack of tooling for producers. The platform addresses these by treating embedding design as the first station and the single source of truth, providing asset‑assisted design, code templates for VSCode and other editors, type checking, and automated testing with one‑click report generation.

Embedding Testing

Testing leverages design‑time rules to automatically validate type, range, and mandatory fields, generating reports that can be sent to developers or data analysts for review.

Embedding Stock Governance

Governance of existing events tackles SLA, cost, compliance, and data quality. Key observations: not all data is important, not all data is useful, and not all data remains compliant. Governance layers include user, statistics, identification, execution, and link layers, each addressing specific needs such as automated usefulness detection, cost accounting, real‑time decision making, end‑to‑end pipeline assurance, and efficient topology solutions.

Embedding Grading & Useless Event Identification

Bloodline extraction differs between offline (point‑to‑point) and real‑time (event‑to‑table) contexts. The platform performs offline SQL parsing, real‑time lineage tracking, instant analysis integration, and recommendation system decoupling. Grading focuses on performance events with tailored SLA and TTL configurations.

Embedding Link Solution

Users—especially non‑technical analysts—need clear insight into required data, its source, and its downstream usage (real‑time reports, behavior analysis, recommendation). Challenges include stability under massive data volumes, low‑latency processing, and graded data lifecycle management.

Data Ingestion : Full‑stack SDKs with built‑in governance, client‑side filtering, and cost‑saving edge computation.

Data Collection : HTTP interfaces feed events into message queues, where events are aggregated into Applog records for real‑time parsing.

Real‑Time Dynamic Processing Engine

The engine provides fast, dynamic processing without restarts, supporting hot‑loaded Groovy scripts, plugin‑based runtimes (Flink, Pyjstorm, TCE), and incremental rule updates. It uses a simple map model to filter and transform incoming data, caches deserialized JSON objects to avoid repeated parsing, and dynamically reconstructs topology based on source changes, reducing Kafka pressure.

Dynamic UDF compilation, topology reconstruction, and RPC updates enable seamless rule modifications. Incremental updates affect only changed rules, and object caching minimizes deserialization costs.

Q&A Highlights

New events become effective within 2 minutes, meeting SLA commitments.

Resource allocation precedes restarts; incremental restarts affect only a subset of nodes.

Code templates are language‑specific and rarely reused across products.

Data loss is mitigated by client‑side retries, monitoring, and server‑side dirty streams; duplicate reporting is rare due to the engine’s dynamic nature.