When sharding databases, independent auto‑increment IDs can collide; this article explains why the conflict occurs and evaluates five concrete solutions—UUID, MySQL auto‑increment step, segment allocation, Snowflake algorithm, and Meituan Leaf—detailing their mechanisms, trade‑offs, performance, and suitability.
This article explains how Kafka achieves extremely high throughput by using a Reactor‑based non‑blocking I/O model, zero‑copy data transfer, sequential disk writes, memory‑mapped files, sparse indexing, partition load‑balancing, compression, batch processing, and a lock‑free offset design.
This article dissects Guava Cache's source code, explaining its segment‑based locking, data structures, put/get implementations, cleanup and eviction mechanisms, and then contrasts its performance and design choices with the more modern Caffeine cache, highlighting why Guava falls short.
This article explains how Kafka achieves ultra‑high throughput and low latency despite being disk‑based, covering its Reactor I/O network model, zero‑copy techniques, partitioning strategies, segment logs with sparse indexes, sequential disk writes, page cache usage, compression, batch processing, and lock‑free offset management.
This article explains how Elasticsearch transforms simple keyword matching into a high‑performance, scalable search engine by using inverted indexes, term dictionaries, posting lists, term indexes, stored fields, doc values, segments, and distributed node architectures to achieve fast, reliable full‑text search on massive data sets.
Since 2018 Youzan has migrated all online services to Kylin4, addressing long cube rebuilds, single‑point cache, CPU spikes, and throttling gaps by adding batch segment builds, low‑priority concurrency controls, Redis‑based query caching, parquet skew mitigation, range‑query acceleration, and class‑loader optimizations, which together doubled query‑per‑second capacity to 150, cut latency by up to 50 % and reduced CPU usage.
This article examines various distributed ID generation techniques—including UUID, database auto‑increment, segment mode, Redis INCR, Snowflake, Meituan Leaf, Baidu UidGenerator, and Didi TinyID—detailing their principles, advantages, drawbacks, and code examples to help developers select the most suitable solution for their systems.
This article explains where MySQL keeps its data files, how InnoDB organizes tablespaces into segments, extents, pages and rows, details the Compact row format—including variable‑length field length lists, NULL‑value lists, and hidden fields—and clarifies the limits of VARCHAR and row‑overflow handling.
This article describes how to identify and fix Greenplum standby segment failures by inspecting segment status, comparing missing files, generating a recovery configuration, executing gprecoverseg commands, and verifying that all segments return to normal operation without service interruption.
This article explains why distributed ID generation is needed, outlines business requirements such as global uniqueness and monotonicity, and compares eight practical schemes—including UUID, database auto‑increment, segment mode, Redis, Snowflake, Baidu UidGenerator, Meituan Leaf, and Didi TinyID—detailing their advantages, disadvantages, and implementation examples.
The article surveys Go‑based distributed ID generation techniques—from UUID, short‑UUID, XID, KSUID, ULID, and Snowflake to database auto‑increment, Redis, ZooKeeper, and segment/leaf approaches—detailing their characteristics, trade‑offs, implementation snippets, and strategies for ordering, uniqueness, and clock‑rollback handling.
This article explains how ElasticSearch achieves near real‑time search by using immutable inverted indexes, segment merging, shard distribution, and a translog for durability, while also offering practical guidance on how to study the system effectively.
This article explains Kafka's storage architecture—including topics, partitions, segments, and their naming rules—along with how data is read, and details the system's reliability features such as ISR/OSR replication, leader election, producer acknowledgment levels, and delivery guarantees.
Elasticsearch routes client write requests to the primary shard, buffers documents in memory, logs operations to a translog, periodically refreshes to create segment files, and uses flush and merge processes to ensure data durability, optimize I/O, and consolidate segments for efficient search.
This article explains Elasticsearch's internal mechanisms for writing, reading, and searching data, covering the roles of coordinating nodes, primary and replica shards, buffers, translog, segment files, refresh cycles, commit and flush operations, as well as Lucene's inverted index and how deletions and updates are handled.
This article explains how Kafka partitions data into segments, stores each segment as paired index and log files, and uses sparse indexing to enable efficient queries, illustrating the process with examples and diagrams of segment layout and offset lookup.
This article explains Kafka’s workflow, detailing how topics, partitions, and segments are organized, the structure of index and log files, message composition, offset-based retrieval, and the overall data directory layout, providing a comprehensive overview of Kafka’s storage architecture.
This article explains Kafka's internal storage architecture—including topics, partitions, segments, .log and .index files—how data is read, and the various reliability mechanisms such as ISR/OSR, LEO/HW, producer acknowledgment levels, leader election strategies, and delivery semantics.
The article explains how Elasticsearch’s use of Lucene’s flush mechanism, concurrent shard writes, and IndexWriter buffering lead to an excess of small segments, outlines the flush conditions, and offers guidance on managing write concurrency for better performance.
This article reviews various distributed ID generation strategies—including database auto‑increment tables, dual‑master MySQL setups, segment allocation, the Snowflake algorithm, and Redis INCR—detailing their designs, SQL examples, scalability trade‑offs, and open‑source implementations.
This article explains how Oracle's logical structures—blocks, segments, tablespaces, and rowids—affect SQL performance, presents detailed experiments on block capacity, row migration, segment handling, tablespace behavior, and provides practical optimization guidelines based on observed logical read reductions.
Kafka stores each topic as partitions that are divided into sequential segment files containing paired .log data and .index files, using global offsets and sparse memory‑mapped indexes to enable fast offset‑based lookups, efficient deletions, and minimal disk I/O in real‑world deployments.
