Exploring Open Source Big Data and AI Technologies
This article explains Spark's memory model, identifies common driver and executor OOM scenarios, and provides detailed configuration and code‑level recommendations—including memory‑related parameters and shuffle‑tuning options—to prevent and resolve out‑of‑memory failures in Spark applications.
This article details Baicaowei's migration from an IDC‑hosted Hadoop cluster to a cloud‑native data lake on Alibaba Cloud, outlining the business drivers, pain points of the legacy platform, architectural goals, design principles, solution selection, implementation steps, and future outlook for the new big‑data ecosystem.
This article provides a comprehensive overview of Snowflake and Delta Lake, detailing their cloud‑native architectures, storage‑compute separation, transaction support, performance optimizations, and a side‑by‑side comparison of their capabilities, openness, and real‑time data handling.
This article provides a detailed introduction to FlinkSQL, covering its background, the Table API, required dependencies, differences between old and Blink planners, various API usage patterns, connector configurations for CSV, Kafka, Elasticsearch, MySQL, and how to convert between DataStream and Table in Flink's unified batch‑stream processing model.
This article presents ByteDance’s ByteLake, a real‑time data lake platform built on Apache Hudi, covering Hudi fundamentals, ByteLake’s use cases, the platform’s architectural optimizations, new features such as a commit‑based metastore and bucket indexing, and future roadmap plans.
This article explains Apache Hudi's two table formats—Copy On Write and Merge On Read—by introducing key terminology, describing their file structures and versioning, comparing write and read latency, I/O cost, and write amplification, and concluding with guidance on choosing the appropriate format.
This article explains Kafka's internal topics, preferred replicas, partition assignment processes, log directory layout, index files, offset and timestamp lookup, log retention and compaction policies, storage architecture, delayed operations, controller role, legacy consumer design flaws, rebalance workflow, and producer idempotence, providing a comprehensive overview of Kafka's backend architecture.
This article shares practical insights on why engineers and leaders must conduct code reviews, identifies common causes of poor code such as duplication and premature decisions, and presents concrete design principles—including simplicity, composition, transparency, and disciplined logging—illustrated with Go code examples to help teams build maintainable, high‑quality backend systems.
This article presents a detailed performance test of HBase 2.x write throughput on a five‑node SSD cluster, identifies latency spikes caused by MemStore flush and ConcurrentSkipListMap size() overhead, and demonstrates how fixing the bug and applying in‑memory compaction dramatically reduces P999 and P9999 latency while preserving throughput.
Apache Kafka 3.0 introduces major enhancements including KRaft consensus, deprecation of Java 8 and Scala 2.12 support, stronger producer guarantees, updated APIs, improved Kafka Connect, MirrorMaker 2 flexibility, and numerous KIP-driven feature upgrades, marking a significant step forward for the distributed streaming platform.
