Building Real-Time Wide Tables with Partial-Update Using Apache Paimon for NetEase News Recommendation

Background: NetEase News, a leading Chinese media portal, needed a more flexible and timely data‑warehouse for its recommendation system, which serves personalized content across headlines, videos, comments, and circles. The existing architecture could no longer meet the growing diversity and complexity of data processing requirements.

Recommendation scenarios: Two business patterns exist – (1) deduplication, where each user‑device (devid) should receive a unique article (docid) only once, using devid+docid as a primary key; (2) non‑deduplication, where paid or experimental content may be recommended multiple times, requiring a composite key devid+docid+rec_time.

Original pipeline: Recommendation data (rec) and user‑behavior data (sdk) were written to HDFS via an internal datastream tool, then joined offline with Spark on an hourly basis, resulting in a visibility delay of H+1 hour and sub‑optimal data quality.

New solution: The team rebuilt the pipeline with Flink and Apache Paimon, enabling near‑real‑time writes and partial‑update capabilities. This reduced the wide‑table latency from hour‑level to minute‑level (depending on Flink checkpoint intervals) and streamlined the processing chain, saving storage and compute resources.

Deduplication implementation: The primary key remains devid+docid. Rec and sdk streams are written to Paimon tables, and a left‑join is performed in Flink, allowing the wide table to be updated incrementally. The approach improves timeliness and simplifies the data flow.

Non‑deduplication implementation: Because rec_time is unavailable in the sdk stream, the latest rec_time for each devid+docid is cached in Flink map‑state. SDK events retrieve this timestamp from the cache to construct the full key, tolerating occasional inaccuracies under high load.

Overall benefits: Introducing Apache Paimon achieved (1) real‑time data visibility (minute‑level), (2) reduced resource consumption and cost, and (3) a simpler, more maintainable architecture.

Additional optimization – recommendation digitization: The team applied Paimon's native aggregation to replace a heavy hourly batch process that summed metrics across dimensions for 20k+ QPS, achieving lower latency and cost while delivering hourly DWS tables for downstream analysis.

Future plans: Implement data‑eviction policies for partially‑updated wide tables to discard irrelevant rows, and leverage Paimon's cross‑partition update feature to eliminate data drift at partition boundaries.