How Zhihu Evolved Its Real-Time Data Warehouse Architecture

Real‑Time Data Warehouse Overview

Data intelligence requires a solid data‑warehouse foundation, and timely data feedback is crucial for rapid product iteration. The article outlines Zhihu's real‑time warehouse practice, covering version 1.0 (ETL real‑time with Spark Streaming) and version 2.0 (layered architecture with Flink Streaming), plus future directions such as Streaming‑SQL platformization, metadata management, and automated result validation.

Version 1.0 Architecture

The 1.0 system processes traffic logs in real time but does not compute metrics. Its three‑stage pipeline consists of:

Data collection: SDKs send logs to a Log Collector Server, which forwards them to Kafka.

ETL: raw logs are cleaned and split into real‑time and batch streams, then imported into Druid.

Visualization: Druid computes metrics and a web server renders them.

Because real‑time streams are less stable than batch streams, a Lambda architecture is used, separating Streaming ETL (Spark Streaming) and Batch ETL.

Framework Selection

In early 2016, Storm and Spark Streaming were the main options. Storm offers pure streaming but lower throughput; Spark Streaming uses micro‑batches, providing higher throughput and a richer ecosystem. Zhihu chose Spark Streaming to meet log volume and initial latency requirements.

Data Correctness

Spark Streaming guarantees end‑to‑end exactly‑once only if downstream is idempotent and upstream can replay. Zhihu achieved at‑least‑once in Spark and added downstream deduplication logic to obtain global exactly‑once.

Common Streaming ETL Logic

All event data share a single Proto Buffer schema:

message LogEntry {
  optional BaseInfo base = 1;
  optional DetailInfo detail = 2;
  optional ExtraInfo extra = 3;
}

The schema defines three parts:

BaseInfo : user, client, timestamp, network information.

DetailInfo : view hierarchy for locating the user.

ExtraInfo : business‑specific fields.

ETL logic is split into generic (BaseInfo & DetailInfo) and non‑generic (ExtraInfo) parts. Three generic logics are highlighted:

Dynamic Streaming configuration via broadcast variables with TTL, allowing metadata changes without restarting jobs.

UTM parameter parsing to attribute traffic to search engines, campaigns, and sharing platforms.

New/old user identification using a two‑layer cache (Thread‑Local + Redis) and HBase persistence; only ~1 % of 260 k/s requests hit HBase.

Spark Streaming Stability Practices

Prefer Direct Kafka mode over Receiver mode to avoid checkpoint‑related restarts and NameNode jitter.

Ensure Yarn allocates sufficient resources; insufficient resources cause frequent executor loss and slowdowns.

Throttle Kafka consumption with StreamingConf to prevent downstream Kafka overload after long downtimes.

Use a Supervisor to automatically restart the Driver when the Spark Streaming job crashes.

Batch ETL and Lambda Architecture

Batch ETL handles data landing, offline processing, and bulk import to Druid. Zhihu built a custom MapReduce Batch Loader for Kafka PB data landing and deduplication, a Repair ETL job for fixing lost or erroneous real‑time data, and extended Druid's import plugin for ProtoParquet files.

Repair ETL

Two Kafka topics store raw logs and post‑ETL logs respectively. Repair ETL runs only when real‑time ETL loses data or encounters logic errors.

Batch Load to HDFS

BatchLoader lands PB data to HDFS, deduplicates, supports multi‑directory partitioning, replay, and self‑dependency management. It has processed over 40 Kafka topics.

Batch Load to Druid

Tranquility streams data into Druid with a time window; late data beyond the window is dropped, so an offline MapReduce job periodically re‑imports missed data.

Shortcomings of Version 1.0

All traffic shares a single Kafka topic, causing duplicate consumption and excessive Kafka outbound traffic.

Druid handles all metric calculations, leading to stability degradation as data volume grows.

Shared traffic source prevents business‑level data isolation and cost accounting.

Version 2.0 Architecture

To address the above issues, Zhihu introduced a layered design:

Raw Layer : collects traffic logs and Binlog changes from multiple apps into separate Kafka topics.

Detail Layer : Streaming ETL (now Flink) processes raw data into structured records.

Summary Layer : aggregates detail data into wide tables; traffic is automatically split per business using a Streaming Proxy that joins with metadata.

Metric Aggregation Layer : stores real‑time metrics in HBase (high‑frequency appends) and Redis (high‑OPS updates).

The metric system integrates with a visualization platform through a defined workflow: requirement documentation → data‑source creation (HBase table + metric column) → dimension table binding → real‑time application writes metrics → business configures real‑time reports.

Application Layer

Three main uses:

Real‑time dashboards powered by the metric aggregation layer.

Recommendation algorithms consuming detail‑summary data.

Tranquility feeding Druid for ad‑hoc multi‑dimensional analysis.

Technical Implementation of Version 2.0

Flink replaces Spark Streaming as the core computation engine for the metric aggregation layer. Advantages include lower latency, exactly‑once semantics, Streaming SQL support, state management, rich windowing, and CEP. Flink Streaming SQL is adopted for platformization, offering high development efficiency and low dimensional cost, though it has syntax differences from Hive SQL and fewer built‑in UDFs.

Progress Achieved with Version 2.0

Traffic splitting serves 14+ business lines, reducing Kafka outbound traffic by an order of magnitude.

Core real‑time reports are stable; over 40 dashboards are now reliably generated.

Ad‑hoc query stability improved as Druid focuses on interactive queries while the warehouse handles metric computation.

Cost accounting per business is possible because each business owns independent data sources and real‑time programs.

Future Outlook

Platformize Streaming SQL so jobs are submitted as SQL files instead of Maven‑packaged JARs.

Systematize real‑time metadata management to lower data‑usage costs.

Automate result validation by comparing Hive and Flink SQL outputs for the same data sources.