Tag

data lake

1 views collected around this technical thread.

iQIYI Technical Product Team

Aug 25, 2023 · Big Data

Venus Log Platform Architecture Evolution: From ELK to Data Lake

The Venus log platform at iQiyi migrated from an ElasticSearch‑Kibana architecture to an Iceberg‑based data lake with Trino, cutting storage and compute costs by over 70%, boosting stability by 85%, and efficiently supporting billions of daily logs through write‑heavy, low‑query workloads.

ElasticsearchTrinoarchitecture evolution

0 likes · 22 min read

Venus Log Platform Architecture Evolution: From ELK to Data Lake

iQIYI Technical Product Team

Jun 30, 2023 · Big Data

Advertising Data Lake Architecture and Real-time Optimizations

By replacing the costly Lambda architecture with a unified data‑lake built on Iceberg and Flink CDC, the advertising team achieved minute‑level latency, strong consistency, and lower storage expenses, cutting end‑to‑end processing times from hours to a few minutes across budgeting, warehousing, OLAP and ETL workloads.

Flinkadvertisingbig data

0 likes · 13 min read

Advertising Data Lake Architecture and Real-time Optimizations