Focused on technical sharing in the data field, covering a tech stack that includes Hadoop, Spark, Flink, Kafka, Fluss, Paimon, Iceberg, StarRocks, ClickHouse, ES, Milvus, and more. Welcome to follow.
This article explains Paimon's LSM-based storage model, detailing its four-tier architecture (MemTable, Immutable MemTable, L0, L1+), write/read paths, Merge-On-Read with loser-tree K-way merge, four merge engines, compaction triggers and modes, changelog production, and performance tuning practices for lakehouse workloads.
This official technical reference details Paimon's file-level index architecture, covering BloomFilter, Bitmap V2, and Range Bitmap indexes with binary storage formats, configuration parameters, hash rules, supported data types, and SQL optimization examples for equality, range, AND/OR, and TOPN queries.
This comprehensive guide covers Bitmap index internals, cardinality thresholds, index replacement rules, and layered optimization combining partition pruning, BloomFilter, and Range-Bitmap for StarRocks querying Paimon tables, with official binary formats, production thresholds, and real-world case studies.
This article analyzes Paimon Bloom Index internals, demonstrates a real-world 25x query acceleration (38s to 1.5s) on a 400GB partitioned table, explains false positive rates, optimal use cases, and provides production-ready configuration patterns for StarRocks+Paimon lakehouse architectures.
This article analyzes how Spark and Flink should divide responsibilities in lakehouse architectures by evaluating three core dimensions—data latency, computation accuracy, and resource cost—arguing that mature batch-stream integration balances business needs against technical overhead rather than pursuing universal real-time processing.
This article details the industry-standard dual pattern of file body storage and metadata registration for unstructured data lake ingestion, covering a three-layer architecture, batch and real-time implementation steps using Paimon and Flink, AI-specific optimizations like label standardization and format conversion, and key operational considerations for permissions, versioning, and cost control.
A big data expert from a large tech company builds a full Hadoop/Spark/CDH stack for a traditional enterprise with only hundreds of thousands of daily records, causing high maintenance costs, half-hour query delays, and eventual departure; the case underscores the importance of matching technology to business scale, with StarRocks proposed as a lightweight alternative.
This guide walks through a systematic troubleshooting process for Elasticsearch yellow cluster status, covering how to diagnose unassigned replica shards using cluster health, shard allocation, and allocation explain APIs, then resolves the five most common causes: insufficient nodes, disk watermarks, disabled allocation, node failures, and oversized shards, with commands for both curl and Kibana Dev Tools.
This article details a comprehensive root cause analysis system for Alibaba Cloud big data platforms, covering a five-layer architecture, three-dimensional analysis (cluster health, task execution, data pipelines), knowledge graph-driven automation, and AI-enhanced future directions to shift from reactive firefighting to proactive defense.
This guide explains why StarRocks temporary partitions block DDL operations due to metadata consistency, locking, and atomicity constraints, and provides step-by-step commands for viewing partitions via SHOW TEMPORARY PARTITIONS and information_schema, plus safe deletion using ALTER TABLE DROP TEMPORARY PARTITION with critical precautions.
