A popular big-data architecture channel with over 100,000 developers. Publishes articles on Spark, Hadoop, Flink, Kafka and more. Visit the Past Memory Big Data blog at https://www.iteblog.com. Search "Past Memory" on Google or Baidu.
The article details how a gaming data‑reporting pipeline uses Redis lists, sets and Lua scripts to build a pseudo‑message queue that provides per‑game message grouping, fixed‑length queues, and batch consumption while meeting strict timeliness and throughput requirements.
This guide walks readers through the complete process of joining an Apache top‑level project—using Apache Doris as an example—including reading the README, joining mailing lists and chat groups, finding a first‑issue, forking the repository, making code changes, submitting a pull request, and passing community review.
The 2022 Enterprise BI Platform White Paper analyzes digital transformation challenges for large enterprises, introduces a Five‑Force BI model, presents case studies from a Fortune‑500 bank, a beverage chain, and a top internet firm, and offers a downloadable full report.
The article explains why event‑tracking data needs governance, outlines a full‑link governance methodology, describes the organizational setup, and details the features of JD Tech’s one‑stop tracking management platform, including metadata unification, one‑click validation, real‑time dashboards, visualization tools, and H5‑native data integration.
Pinterest’s Monarch batch‑processing platform, built on over 17 k YARN nodes in AWS, was upgraded from Hadoop 2.7.1 to 2.10.0 using a phased, cluster‑by‑cluster strategy that balanced minimal downtime, extensive validation, and custom patches to handle compatibility and dependency issues.
On August 10, 2022 Apache DolphinScheduler 3.0.0 was officially released, introducing a brand‑new Vue3‑based UI that is dozens of times faster, extensive AWS support, custom time‑zone handling, task groups, native data‑quality checks, service splitting for container‑native deployment, numerous new task types, Python API enhancements, and a long list of bug fixes and documentation updates.
The article examines the open‑source release of Delta 2.0, compares its features and benchmark results with Iceberg and Hudi, discusses the core capabilities required by enterprises for a lakehouse architecture, and introduces the Arctic project as a multi‑engine streaming lake service.
This article provides a comprehensive overview of the big data ecosystem, detailing nine core technology categories—from data collection and storage to computation, analysis, scheduling, and underlying infrastructure—along with tool comparisons, selection guidelines to help readers quickly build a complete big data knowledge system.
The article compares Data Fabric and Data Mesh as modern data‑architecture approaches, explains their technical and organizational differences, discusses the ongoing debate between data lakes, warehouses, and lakehouses, and highlights how each option fits varying data‑type and usage scenarios.
