kvrocks Boosts HuoLala Trace Storage: Overcoming Compaction to Reach 70k QPS

Introduction

In high‑traffic trace scenarios, a high‑performance, large‑capacity, persistent NoSQL store becomes essential. kvrocks, built on RocksDB and compatible with the Redis protocol, was chosen for secondary development due to its lightweight design and active community.

Trace Overview

Distributed tracing is a key part of observability (Metric & Trace & Log). HuoLala’s trace platform, based on Apache SkyWalking, evolved to version 3.0, serving over 1,000 applications.

Typical trace data forms a directed acyclic graph and is stored as key‑value pairs (TraceID → full Span data) in distributed NoSQL stores such as ES or HBase.

kvrocks Overview

kvrocks Basics

kvrocks uses reuse_port multi‑threaded listening, processes Redis commands, and maps them to RocksDB key‑value entries; each kvrocks node hosts a single RocksDB instance.

RocksDB Basics

RocksDB is an LSM‑Tree storage engine that buffers writes in memory and flushes them to disk in batches, reducing I/O compared with InnoDB. Compaction merges SST files into hierarchical levels.

Memtable

Implemented as a skip‑list, Memtables hold writes until they become immutable and are flushed to disk; their size and count can be tuned for performance.

Compaction Strategies

Leveled Compaction organizes files into levels (L0, L1, …) to balance write, read, and space amplification.

Universal Compaction groups files into tiers, reducing write amplification at the cost of higher read and space amplification.

FIFO Compaction maintains a single layer and discards the oldest files when storage limits are reached, offering minimal write amplification and no space amplification.

HuoLala Trace Experiments

The trace workload is write‑heavy (≈600 k write QPS, ~20 k read QPS) with short data TTL.

Leveled Strategy Tests

Initial configuration (256 MiB Memtables, 6 instances, 12 background threads) caused “Compaction pending” and could not sustain the write pressure. Increasing Memtable count and threads delayed but did not eliminate the issue.

Universal Strategy Tests

Switching to Universal compaction eliminated pending compactions over three days of heavy load, though occasional latency spikes appeared. Root causes identified were:

Lock contention on hset meta‑data updates.

Single‑threaded write path for memory and log flushing.

Excessive file scans during reads.

Mitigations included removing unnecessary meta‑data, enabling unordered_write, and prefixing keys with timestamps to reduce range overlap.

Result: the system handled the write load, but space amplification rose to ~1.7×, increasing cost.

FIFO Strategy Tests

Adopting FIFO compaction with 1 GiB Memtables and disabling compaction yielded perfect fit for the trace use‑case: no write amplification, automatic TTL‑based deletion, and acceptable read performance for the workload.

Scaling and Migration

Data migration uses snapshot‑based full copy followed by WAL‑based incremental sync. For TB‑scale data, expanding WAL size is required. A request‑forwarding scheme based on timestamped keys was chosen to support cluster topology changes without heavy data movement.

Conclusion

After migrating trace storage to kvrocks, HuoLala has operated fault‑free for three months, handling near‑hundred terabytes of data, peak QPS of 70 k, and achieving a 55% cost reduction compared with HBase. kvrocks meets the trace storage requirements with superior performance and economics.