This article explains the evolution of Vivo's real‑time recommendation feature‑concatenation architecture, compares hour‑level, Redis‑streaming and RocksDB state‑backend solutions, and details the memory, performance, startup and HDFS RPC problems encountered along with the concrete fixes applied.
This article explains how overusing RocksDB's DeleteRange for bulk file deletions can cause severe get latency spikes and uncontrolled memory growth, analyzes the underlying range tombstone mechanisms, and shares a practical optimization that replaces DeleteRange with regular deletes.
The article recounts a series of three OOM incidents in a production service that combines Paimon data lake and RocksDB via an SDK, detailing the step‑by‑step investigation, the discovery of excessive bucket‑driven thread creation and off‑heap memory leaks, and the final mitigation measures that restored stability.
This article provides an in‑depth technical exploration of RocketMQ 5.0's ConsumeQueue component, explaining why it is needed, its design principles, the traditional file‑based implementation, and a performance‑focused RocksDB‑based redesign, complete with code excerpts and implementation details.
LinkedIn’s message search system stores messages in RocksDB, builds Lucene inverted indexes on demand, partitions them by user, keeps indexes in memory, and uses a coordinator with D2/Zookeeper for node routing, enabling rapid, cost‑effective searches while minimizing write overhead.
The article examines a production RocksDB memory‑usage problem where two instances consumed 59 GB on a 32‑CPU, 64‑GB server, identifies glibc ptmalloc’s unreclaimed free memory as the main cause, and shows that switching to jemalloc cuts usage by roughly 25 % while improving I/O and CPU efficiency.
This article introduces Pika 3.5.x’s major enhancements—including a new sync mechanism, expanded Redis command support, RocksDB upgrades, BlobDB, containerization, observability tools, ACL and transaction support—while providing a step‑by‑step upgrade procedure, optional data synchronization, and two rollback strategies for safe migration.
This article examines Rockset’s cloud‑native real‑time analytics database, detailing its document‑oriented data model, RocksDB‑Cloud storage engine, compute‑storage separation, sharding, converged indexing, query processing pipeline, and the implications of OpenAI’s recent acquisition for the broader database ecosystem.
This article explains the design and implementation of ROR (Redis‑On‑Rocks), a Redis extension that adds hot‑cold data swapping using RocksDB to dramatically reduce memory costs, improve performance, and remain compatible with Redis commands, while detailing benchmarks, concurrency control, and future roadmap.
In large‑scale Flink double‑stream joins, terabyte‑sized RocksDB state caused severe compaction latency and CPU spikes, but enabling RocksDB BlobDB KV‑separation (and an inner‑compaction patch) dramatically shrank SST files, reduced read/write latencies to sub‑millisecond levels, and cut CPU spikes by about half.
The article explains RocksDB’s LSM‑based architecture, column‑family isolation, and snapshot features, and shows how Vivo’s VPUSH mapping service uses these capabilities to store billions of registerId‑to‑ClientId mappings with high‑concurrency, low‑cost, fault‑tolerant performance across multiple replicated servers.
The article explains RocksDB’s LSM‑based architecture, column‑family isolation, and snapshot features, and shows how Vivo’s VPUSH MappingTransformServer uses these capabilities with C++ code to store billions of registerId‑to‑ClientId mappings across multiple replicated servers for high‑concurrency, low‑latency, and fast service expansion.
Redis‑On‑Rocks (ROR) extends the Redis codebase with RocksDB to provide cold‑hot data tiering, reducing memory costs by about two‑thirds while maintaining low latency and high throughput, and the article details its design, implementation, benchmarking against Redis‑on‑Flash, and production experience.
This article explains how to optimize Flink’s RocksDB state backend for large‑scale streaming jobs, covering state types, enabling latency tracking, incremental checkpoints, predefined options, and advanced memory and thread settings, with practical configuration examples and performance comparisons.
This article explains how Bilibili’s KV storage system evolved from early Redis/Memcache solutions to a custom distributed KV architecture, detailing design goals, architecture components, shard management, Raft replication, multi‑active disaster recovery, and the operational challenges solved to support massive traffic growth.
Apache Flink 1.17 introduces a suite of batch and streaming enhancements—including a new Streaming Warehouse API, significant TPC‑DS performance boosts, adaptive batch scheduling, improved checkpointing, expanded SQL capabilities, Hive connector upgrades, and broader filesystem support—while also delivering upgrades to FRocksDB, Calcite, and the token framework to strengthen its position as a leading unified data‑processing engine.
This article outlines the latest RocksDB release updates, including disabled checksum verification options, removed Lite and deprecated APIs, default shared library builds, new compression filter and MultiGetEntity APIs, and various bug fixes such as concurrency issues and async I/O error handling.
By manually modifying a few hundred lines and using an automated script, we transformed the multithreaded RocksDB storage engine into a coroutine‑based version with PhotonLibOS, achieving near‑identical functionality and, in heavy I/O and high‑concurrency scenarios, up to double the throughput compared to the original.
This article explains how UCloud’s UKV-Meta leverages a customized URocksDB storage engine, addresses RocksDB’s durability and read‑performance drawbacks with a Share‑Nothing architecture, introduces hotspot splitting, hot‑standby replication, and seamless data migration to achieve high availability and scalability in cloud object storage.
The new RocksDB 7.5.3 release dramatically reduces write‑stall time, lock contention and CPU usage while improving OPS and latency compared with 6.2.9.x, as shown by detailed memtier benchmark tests on a 4‑CPU, 32 GiB VM with NVMe storage.
This article explains the principles of Open-Channel SSDs, their evolution into Zoned Namespace (ZNS) technology, compares performance of a commercial ZNS drive with traditional SSDs, and explores how databases like RocksDB and the ZenFS file system can exploit ZNS for higher efficiency and lower latency.
This article details the design and implementation of a high‑performance data‑consistency verification platform that supports both real‑time and 10‑minute offline checks on tens of billions of records, covering background challenges, system architecture, processing pipelines, RocksDB integration, performance results, and practical lessons learned.
This article explains how RocketMQ implements scheduled and delayed messages using fixed delay levels, the internal storage flow, code examples for setting delay levels, and advanced techniques such as RocksDB integration to achieve arbitrary time precision for delayed delivery.
This article explains how Apache BookKeeper's garbage collection works, detailing minor and major GC triggers, compression strategies, entry log size management, usage calculations, and provides a step‑by‑step code analysis of the GarbageCollectorThread implementation.
This article presents a detailed guide for optimizing large‑scale Apache Flink streaming jobs on the JD Real‑Time Computing platform, covering TaskManager memory model tuning, network stack configuration, RocksDB state management, checkpoint strategies, and additional performance tips with practical examples and calculations.
This article details Ctrip's adoption of Nebula Graph, covering the motivations, distributed architecture, three deployment patterns, Kubernetes‑based operators, client session management, blue‑green read/write splitting, extensive performance tuning, and future roadmap for their production graph database platform.
WLock is an open‑source distributed lock service built on the WPaxos consensus algorithm and RocksDB storage, offering multiple lock types, flexible acquisition modes, high concurrency optimizations, and strong reliability and throughput for coordinating shared resources in distributed systems.
This article surveys common delayed‑message implementations—including external storage, RocksDB, Redis, and open‑source MQs like RocketMQ, Pulsar, and QMQ—detailing their architectures, advantages, disadvantages, and practical considerations for distributed asynchronous messaging systems.
This article surveys common delayed‑message solutions in distributed asynchronous messaging, evaluates implementations based on external storage, databases, RocksDB, Redis, and open‑source MQs like RocketMQ, Pulsar and QMQ, and discusses their advantages, drawbacks, and practical design considerations.
This article explains the shortcomings of HDFS at large scale, describes the Federation and Scaling approaches, and details how Apache Ozone redesigns metadata storage, introduces container abstraction, object semantics, and new features such as optimized OM, streaming writes, erasure coding, and RocksDB consolidation to improve scalability and performance.
This article reviews common delayed‑message implementation approaches—including external storage, RocksDB, Redis, and built‑in solutions in open‑source MQs like RocketMQ, Pulsar, and QMQ—detailing their architectures, advantages, drawbacks, and practical considerations for distributed systems.
This article analyzes a real‑world migration from MongoDB to the open‑source Pika KV store, demonstrating how adjusting the max‑write‑buffer‑size parameter dramatically improves disk I/O characteristics and raises query‑per‑second throughput from 3 K to 40 K.
In just 13 days Bilibili transformed a simple MySQL‑based S3 prototype into the BOSS distributed object storage system by separating metadata and data, adding an RPC abstraction layer, implementing two‑level sharding, switching to RocksDB, and deploying a three‑replica, multi‑zone high‑availability architecture.
This article surveys common delayed message implementations—including external storage, RocksDB, Redis, and open‑source MQs like RocketMQ, Pulsar, and QMQ—analyzing their architectures, advantages, drawbacks, and practical considerations for building reliable distributed asynchronous messaging systems.
The article surveys common delayed‑message solutions in distributed MQ systems—including external storage, RocksDB, Redis, thread‑based scanning, and the designs of RocketMQ, Pulsar, and QMQ—detailing their architectures, advantages, and drawbacks for backend developers.
This article presents a detailed evaluation of Intel® Optane™ Persistent Memory, including benchmark comparisons with DRAM and NVMe using Redis and RocksDB workloads, analysis of latency and throughput, and step‑by‑step configuration instructions for Memory Mode, App Direct, and KMEM DAX.
To meet Cloud Music’s massive algorithm‑feature KV storage needs, the team built RDB—a RocksDB‑based engine within Tair—adding bulk‑load, dual‑version imports, KV‑separation, in‑place sequence appends and protobuf field updates, cutting storage cost, write amplification and latency while scaling to billions of records and millions of QPS.
This article examines various delayed‑message implementations in distributed message‑queue systems, comparing external‑storage approaches using databases, RocksDB and Redis, and reviewing built‑in solutions in open‑source MQs such as RocketMQ, Pulsar and QMQ, while discussing their advantages, drawbacks and design considerations.
This article examines common delayed (scheduled) message implementations in distributed systems, comparing external storage approaches using databases, RocksDB, and Redis, as well as built-in solutions in open-source MQs like RocketMQ, Pulsar, and QMQ, and discusses their advantages, drawbacks, and design considerations.
This article explains the fundamentals of Log‑Structured Merge (LSM) Trees, covering their write‑ahead log, memtable, SSTable architecture, compaction processes, read/write optimizations, and popular open‑source implementations such as LevelDB, RocksDB, and GoLevelDB.
Percolator implements a two‑phase commit transaction protocol with a client coordinator, a timestamp oracle, and storage (Bigtable or TiKV’s RocksDB), providing snapshot‑isolated ACID semantics via lock, write, and data columns; TiKV adds parallel prewrite, short‑value storage, point‑read shortcuts, calculated commit timestamps, and single‑region one‑phase commits to boost performance while keeping the design simple and scalable, though high contention can cause retries and read‑wait delays.
To meet Baidu Chinese’s need for sub‑200 ms responses on multi‑hop queries across millions of dictionary entities, the team replaced MySQL with the open‑source HugeGraph graph database backed by RocksDB, deploying a multi‑master, REST‑enabled architecture with caching, bulk loading, and a data‑intervention platform to ensure fast, reliable traversal of semantic relationships.
This article introduces TiDB’s hybrid transactional‑analytical processing capabilities, explains TiKV’s region‑based storage built on RocksDB, and provides a detailed overview of RocksDB and LevelDB architectures, including their data structures, write‑read flows, and key optimizations for modern distributed database systems.
Tongcheng Yilong’s BaikalDB deployment combines high‑availability multi‑Raft HA, HTAP support, and share‑nothing scalability to deliver over 72K TPS OLTP and ten‑fold faster OLAP queries while cutting operational costs up to a hundredfold through dual‑center, columnar storage and cloud‑native elasticity.
This article examines the requirements of commercial advertising systems for data storage, traces the evolution from single‑node MySQL to BaikalDB’s cloud‑native, MySQL‑compatible distributed architecture, and details its storage, compute, and scheduling designs, highlighting key features such as Raft replication, RocksDB storage, and hybrid OLTP/OLAP support.
This article analyzes the evolution of Baidu's advertising data storage, detailing the business-driven requirements, the design and development of the BaikalDB distributed database, its architecture across storage, compute and scheduling layers, key features such as Raft replication and multi‑index support, and the lessons learned for building cloud‑native, high‑performance databases.
This article walks through creating a lightweight Redis‑style key/value service called kedis by integrating Facebook's RocksDB storage engine and the RestExpress HTTP container, showing how to set up the server, define API routes, and perform basic put/get operations via HTTP requests.
The article introduces the pain points of using Redis at Tencent IEG, explains the three Tendis product editions, and provides an in‑depth description of the hybrid storage version’s architecture, components, version control, cold‑hot data interaction, scaling mechanisms, and the stateless Redis‑sync layer.
Tendis, an open‑source distributed KV store developed by Tencent, combines full Redis protocol compatibility with RocksDB persistence, a decentralized gossip‑based cluster architecture, horizontal scalability to thousands of nodes, and cost‑effective hot‑cold data mixing, making it suitable for large‑scale, high‑throughput workloads.
This article explains YugabyteDB's two‑layer logical architecture, its tablet‑based distributed storage built on Raft groups, the RocksDB‑backed local DocDB, and how it implements distributed transactions using Hybrid Logical Clocks, two‑phase commit, and MVCC, while comparing it with TiDB, CockroachDB and other rivals.
This article provides a comprehensive technical overview of YugabyteDB, covering its two‑layer logical architecture, tablet‑based distributed storage with Raft groups, RocksDB‑backed local storage design, hybrid hash‑range partitioning, and the MVCC‑based two‑phase‑commit transaction model using Hybrid Logical Clocks.
TXRocks is a transactional storage engine based on RocksDB that offers significant space savings compared to InnoDB while maintaining comparable performance, with optimizations including sum operator pushdown and AEP-based secondary caching to address RocksDB's read performance limitations.
Tendis is a Tencent‑developed distributed key‑value database that fully supports Redis protocols, leverages RocksDB for persistent storage, and uses a decentralized gossip‑based cluster architecture to deliver large‑capacity, low‑cost, and high‑availability solutions for warm‑cold data workloads.
Since 2019 NetEase Media has migrated several recommendation and account services from RDS to MyRocks, cutting disk usage by up to 68 % and halving response times while handling 40‑50 k QPS write‑heavy workloads, though the engine lacks partitioning, online DDL, and certain index types, requiring careful workload assessment.
The article explains the background, architecture, data distribution, processing flow, replication mechanisms, and management features of Pika's native distributed cluster, detailing how Etcd, LVS, and RocksDB are used to achieve scalable, persistent Redis-compatible storage with table isolation and flexible slot replication.
This article demonstrates how to create a lightweight key/value store named kedis by integrating Facebook's RocksDB storage engine with the RestExpress HTTP container, providing step‑by‑step instructions, code examples for initializing the database, defining routes, and running the service via HTTP GET requests.
The article starts with a job‑interview anecdote and then provides a comprehensive overview of NoSQL, its history, definitions, comparison with MySQL, practical scenarios, and detailed examinations of key NoSQL projects such as LevelDB, RocksDB, SSDB, and Pika, highlighting their architectures and use cases.
Facing a ten‑fold cost increase for public‑cloud Redis, Ctrip engineers evaluated Redis‑on‑SSD alternatives, chose KVROCKS, performed extensive protocol‑compatible modifications, benchmarked performance on standard and Optane SSDs, and demonstrated substantial cost savings while preserving Redis semantics.
This article explains the design of a data exploration service for large‑scale distributed storage systems, detailing metadata synchronization, index reconstruction, aggregation tables, node‑level statistics, a user console, and the transition from checkpoint‑based snapshots to delta updates using RocksDB WAL in Hadoop Ozone Recon Server.
This article details Ctrip's investigation of SSD‑based KVROCKS as a low‑cost, Redis‑compatible alternative, covering background, solution selection, secondary development steps, extensive performance testing on different SSDs, cost‑saving analysis, encountered pitfalls, and future plans.
This article describes ByteDance’s extensive improvements to the RocksDB storage engine—including LazyBuffer, Adaptive Map‑based lazy compaction, KV separation, adaptive multi‑index support, extreme compression techniques, and hardware acceleration—to reduce amplification, improve performance, and lower costs for large‑scale database workloads.
This article analyzes RocksDB's compaction rate‑limiting source code and presents practical tuning methods—both fixed and auto‑tuned—to mitigate IO spikes in the distributed KV store WTable, improving real‑time read/write latency and stability.
This tutorial demonstrates how to use Facebook’s RocksDB storage engine together with the Netty‑based RestExpress HTTP container to create a lightweight Redis‑style key/value service called kedis, covering project setup, core API implementation, route configuration, building, launching, and basic curl interactions.
This article details how Weibo leverages Redis across billions of requests, describing its massive scale, the challenges of trillion‑level reads/writes, the technical choices and customizations made—including LongSet, HA solutions, multi‑level caching, RocksDB integration—and outlines ongoing capacity and future development strategies.
This article explains how Flink checkpoints persist state, compares the three built‑in state backends (MemoryStateBackend, FsStateBackend, RocksDBStateBackend), discusses their configurations, advantages, limitations, and provides guidance on selecting the appropriate backend for different big‑data streaming scenarios.
FastLoad, Didi’s one‑click DTS platform, accelerates migration of terabyte‑scale offline data into its Fusion storage by using RocksDB’s IngestFile to import SST files directly, cutting a 1 TB load from twelve hours to one, while supporting thousands of daily tasks with 99.99% stability.
This article explains Flink's checkpoint mechanism, its execution flow, common configuration options, and the benefits and considerations of incremental checkpoints using the RocksDB state backend, providing practical code examples and YAML settings for reliable stream processing.
This article explains CB‑SQL's two backup approaches—logical (using DUMP and IMPORT) and physical (using BACKUP and RESTORE)—detailing their mechanisms, supported formats, storage options, performance characteristics, and how they ensure reliable data recovery for large‑scale distributed databases.
The article details how Kuaishou leverages Apache Flink for large‑scale stream processing, describing its application scenarios, cluster sizing, interval join optimization, RocksDB performance challenges, source throttling strategies, JobManager stability, frequent job failures, and platform‑wide improvements.
This article introduces CB‑SQL, a MySQL‑compatible elastic database built on CockroachDB and RocksDB, explains RocksDB’s LSM‑tree design, details key configuration parameters, and describes the read/write/clear/file‑operation optimizations employed by CB‑SQL to achieve high scalability and stability.
This article explains how the WTable distributed key‑value store leverages RocksDB’s LSM‑tree architecture and slot‑based data distribution to redesign its scaling workflow, separating full and incremental data migration to reduce compaction overhead and achieve high‑speed, low‑impact cluster expansion.
This article explains how Apache Flink's State TTL feature works, demonstrates configuring TTL for state size control and automatic cleanup, and details the continuous cleanup mechanisms introduced in Flink 1.8.0 for both heap and RocksDB state backends.
Fusion, Didi’s home‑grown C++ NoSQL database built on RocksDB, powers over 400 services and 1.5 PB of data, then evolved into the NewSQL layer dise with schema, secondary indexes, transactions and MySQL‑compatible binlog, and now guides a next‑gen distributed database project targeting true elastic scaling, distributed transactions and full SQL compatibility.
This article details Didi's evolution from a self‑built NoSQL system called Fusion to a NewSQL solution and outlines the challenges, architectural highlights, data migration mechanisms, and future distributed database design, providing concrete metrics, diagrams, and practical lessons learned.
This article explains how to map relational table rows and various indexes—including primary key, integer, float, string, and composite indexes—into RocksDB key‑value pairs using custom serialization, big‑endian ordering, and encoding functions to enable efficient point, range, and multi‑field queries.
To overcome severe performance degradation of Flink sliding windows over very long time ranges, Youzan engineers applied time‑slicing based on the greatest common divisor of window length and slide step, reducing state writes and timers, which yielded 3‑8× speedups in production.
This article presents thirty practical recommendations for deploying, configuring, and maintaining Pika—a high‑capacity, RocksDB‑backed Redis‑compatible storage system—covering version selection, thread settings, hardware choices, key design, memory management, replication, backup, compaction, security, and monitoring to achieve reliable and high‑performance operation.
Pika 3.0, the open‑source C++ Redis‑compatible storage built on RocksDB, introduces a new Blackwidow engine, enhanced binlog, and optimized server layer, delivering higher interface performance, reduced disk usage, extended key length, full ZSET compatibility, and improved overall efficiency.
This article analyzes the high space efficiency but poor sum performance of TXRocks compared to InnoDB, identifies three main bottlenecks, and details three optimizations—sum push‑down, selective record conversion, and multithreaded execution—that together reduce query latency to less than 5% of InnoDB’s original time.
The article summarizes the 58 Group technical salon where experts compared Meituan‑Dianping's Cellar and 58's WTable distributed KV storage systems, detailing their architectures, improvements, scalability, high‑availability mechanisms, and operational considerations, and concludes with a comparative analysis and preview of the next session.
Pika 3.0, a C++ open‑source Redis‑compatible storage built on RocksDB, introduces a new blackwidow engine, an improved binlog system, and a more efficient server layer, delivering higher performance, reduced disk usage, extended key length limits, and full Redis ZSET compatibility.
This article details the design, architecture, and implementation of ZanKV—a high‑performance, distributed key‑value store that combines RocksDB storage, etcd‑Raft consensus, and a Redis‑compatible protocol, covering data partitioning, namespace isolation, expiration strategies, cross‑datacenter deployment, and performance tuning.
This article provides a comprehensive technical overview of Ceph's storage back‑ends, detailing the evolution from FileStore to NewStore and the latest BlueStore implementation, its architecture, raw‑device management, and performance advantages for distributed storage systems.
Pika 3.0 introduces a new Blackwidow engine, an efficient write‑to‑file binlog, and a streamlined server layer, all designed to boost performance, reduce storage costs, and fully maintain Redis‑compatible functionality for large‑scale deployments.
iQiyi improved OpenStack Swift’s video‑streaming performance by redesigning its storage engine to merge numerous tiny objects into larger files, replacing Swift’s per‑object directory structure with RocksDB‑managed metadata, eliminating file‑ID overhead, using prefix scans for fast listings, and adding lazy deletion, achieving dramatic speedups with only about 3,000 lines of new code.
TXSQL is Tencent Cloud’s MySQL‑based kernel, engineered to handle massive scale, improve stability, performance, and security for diverse industries, featuring optimized redo‑logs, thread‑pool handling, encryption, audit plugins, and parallel replication, with a rigorous release process and future plans for engine‑level batch computation and RocksDB integration.
This article explains the ACID properties, write‑ahead logging, and how RocksDB implements a three‑step write path and a leader‑follower Group Commit mechanism to improve transaction throughput by batching fsync operations.
This article presents an in‑depth overview of Zeppelin, a high‑availability, high‑performance block storage service, covering its motivation, online vs offline storage distinctions, data distribution strategies, centralized meta‑server design, replication policies, RocksDB‑based storage engine, Raft‑based consistency protocol, threading model, client request flow, and fault‑handling mechanisms.
This article explains the internal workings of the SHOW ENGINE ROCKSDB STATUS command in MyRocks, detailing how STATISTICS and BG_THREADS data are collected, the underlying RocksDB APIs, code structures, and how to extend these mechanisms for performance monitoring and debugging.
Since its inception, Facebook has relied on MySQL to handle data from over two billion users, but recent shifts toward NoSQL have led to the development of Apollo—a layered storage system inspired by Paxos, Raft, RocksDB, and custom APIs, aiming to improve scalability, latency, and fault tolerance.
This article examines the limitations of conventional block compression in modern databases, introduces FM-Index and its drawbacks, and explains how Terark's searchable compression—combining CO-Index for keys and PA‑Zip for values—delivers higher compression ratios, faster random reads, and eliminates double‑caching overhead.
This article provides a comprehensive overview of RocksDB, covering its origin, design goals, core architecture components, key features such as APIs, compression strategies, durability mechanisms, backup and replication support, as well as tooling, testing, and performance characteristics.
TiDB’s founder Huang Dongxu recounts the journey of building a MySQL‑compatible, distributed database—detailing early challenges, architectural decisions, the extensive testing, the adoption of Go, Rust, Raft, RocksDB, and the emphasis on metrics, cloud‑native design, and open‑source community collaboration.
This article explains how TiKV implements multi-version concurrency control (MVCC) on top of RocksDB and details its two‑phase commit transaction model, including Prewrite and Commit phases, Percolator‑style optimizations, lock handling, conflict resolution, and garbage‑collection strategies.
The article details Snapdeal's Ads platform architecture, key strategies, infrastructure, requirements, and technologies that enable a highly available, low‑latency backend capable of processing billions of daily requests with a small engineering team.
