A popular big-data architecture channel with over 100,000 developers. Publishes articles on Spark, Hadoop, Flink, Kafka and more. Visit the Past Memory Big Data blog at https://www.iteblog.com. Search "Past Memory" on Google or Baidu.
Uber runs over 20 Presto clusters serving more than 500,000 daily queries, but frequent full GCs and OOMs threatened stability; by analyzing G1GC behavior and adjusting IHOP, heap waste, free space, and young‑gen size on JDK 8 and JDK 11, they cut full GC occurrences by up to 80% and markedly improved overall reliability.
Databricks announced its acquisition of Tabular, the creators of Apache Iceberg, aiming to unify lakehouse formats through Delta Lake UniForm, while highlighting the rise of lakehouse architecture, format fragmentation, and the push toward open data interoperability.
MetaApp replaced ClickHouse with the open‑source cloud‑native warehouse ByConity, achieving more than 50% reduction in resource costs while delivering comparable or faster OLAP query performance across distinct, retention, conversion, and point‑lookup workloads, thanks to compute‑storage separation, read/write isolation, and minute‑level elastic scaling.
ByteHouse, a cloud‑native data warehouse built on ClickHouse, redesigns storage‑compute separation, introduces a new MPP architecture, rule‑based and cost‑based optimizers, exchange runtime filters, and parallelism techniques, delivering 10‑200× faster query performance on TPC‑DS, TPC‑H and SSB benchmarks and boosting point‑lookup QPS to 32,000.
WeChat evolved its data platform from Hadoop to ClickHouse and finally to a StarRocks‑based lakehouse, solving data fragmentation and storage redundancy while achieving sub‑second to minute‑level query latency, cutting storage costs by over 65%, halving operational tasks, and reducing offline job time by two hours across several business lines.
The article summarizes a PrestoCon talk that reviews Meta's year‑long production experience with Prestissimo—a C++ Presto worker built on the Velox execution engine—highlighting its architecture, integration design, performance gains, and lessons for anyone considering Velox‑based vectorization.
This comprehensive 2023 guide covers essential big‑data interview topics, providing detailed explanations and step‑by‑step processes for Hadoop HDFS read/write, YARN, Hive table types and optimizations, Doris architecture and data models, data‑warehouse layers, modeling techniques, quality monitoring, and classic algorithm design questions such as TOP‑K and duplicate detection.
Sichuan Rural Credit Union, with over 5,000 outlets and 3 billion daily transactions, migrated its core and mobile banking systems to OceanBase's native distributed database, cutting card‑issue time from 30 minutes to 5 minutes, loan processing from 3‑5 days to 1‑33 minutes, and saving more than 40% in infrastructure costs.
The article analyzes SQLite's shortcomings in data‑source support, complex calculations, and workflow handling for tiny Java apps, then demonstrates how the open‑source esProc SPL engine offers richer data‑source integration, simpler SQL‑like syntax, powerful calculation capabilities, and built‑in flow control, making it a more suitable lightweight database alternative.
The article analyses SQL’s fundamental shortcomings—its lack of ordered collections, incomplete set semantics, and missing object‑reference support—illustrates these issues with concrete query examples, and argues that the open‑source Structured Process Language (SPL) solves them with true ordering, full collection handling, and native reference mechanisms.
