How TiDB’s New Tiered Storage Architecture Cuts Costs and Boosts Performance

When data volume grows, only about 20 % of rows are hot and need SSD latency; the remaining 80 % are cold and can be stored on cheaper object storage, reducing SSD cost by roughly 50 %.

Core concepts

IA table (Infrequent Access): a table enabled for tiered storage.

Hot data : resides on local SSD with sub‑millisecond latency.

Cold data : stored in remote object storage (S3/OSS).

Segment : the smallest unit TiKV reads from object storage, about 1 MiB.

Tiered storage : can be applied at table or partition level.

Architecture evolution

Classic TiKV runs a single RocksDB LSM‑Tree per node, which leads to resource contention and inflexible scaling.

Multi‑tree parallelism

Each node runs multiple LSM trees in parallel, reducing lock conflicts and providing physical resource isolation between tables or tenants.

S3 as persistent layer

KV files (SST) are persisted to object storage; Raft logs and WAL stay on local disks (optional S3 backup). This design enables fast node recovery from S3, independent scaling of compute and storage, and forms the basis for tiered storage.

Segment caching – reducing cold‑read overhead

Cold data is stored as whole SST files on S3. When a cold read occurs, TiKV downloads only the required Segment (≈1 MiB) instead of the entire file. A tiny local meta file indexes all segments, allowing precise lookup.

Write path : memory → L0 → compaction → cold data finally lands in S3.

Read path : start from MemTable, descend levels; if a needed segment is not cached, fetch it from S3.

Using tiered storage

Option 1 – Whole‑table IA (simplest) :

CREATE TABLE orders (
    id BIGINT PRIMARY KEY,
    user_id BIGINT,
    amount DECIMAL(10,2),
    created_at DATETIME
) STORAGE_CLASS IA;

Existing tables can be altered online: ALTER TABLE orders STORAGE_CLASS IA; Option 2 – Partition‑level IA (recommended) – time‑based partitions store historical data in IA while recent partitions stay hot:

CREATE TABLE orders (
    id BIGINT,
    order_date DATE,
    amount DECIMAL(10,2),
    ...
) PARTITION BY RANGE (order_date) (
    PARTITION p2023 VALUES LESS THAN ('2024-01-01') STORAGE_CLASS IA,
    PARTITION p2024 VALUES LESS THAN ('2025-01-01') STORAGE_CLASS IA,
    PARTITION p2025 VALUES LESS THAN ('2026-01-01') STORAGE_CLASS STANDARD,
    PARTITION p_future VALUES LESS THAN MAXVALUE STORAGE_CLASS STANDARD
);

Benefits of partition‑level IA include precise cost control, predictable performance, strong isolation, and simple operations when adding new partitions.

Feature limits & cold‑read constraints

Indexes follow the table’s IA setting; they cannot be set individually.

Hash/Key partitioning does not support IA.

IA applies only to TiKV row‑store; TiFlash/TiSearch remain on local storage.

Cold‑read bandwidth is throttled to protect the cluster:

Single‑SQL cold‑read throughput ≤ 100 MiB/s (prevents a single query from hogging bandwidth).

Concurrent cold reads ≤ 1 GiB/s (≈10 concurrent queries) to protect other tenants.

TiKV single‑miss load ≤ ~3 MiB (≈3 segments) per miss.

Read amplification

A 100‑byte record may trigger download of three 1‑MiB segments (≈30 000× amplification). Two cases exist:

Beneficial amplification : the segment stays in IA cache, turning future reads hot.

Harmful amplification : large cold scans evict hot data, causing cascading cache misses.

Therefore tiered storage is unsuitable for frequent wide scans.

IA Manager – intelligent cache management

IA Manager automatically detects hot spots and manages the lifecycle of hot/cold data using a two‑layer cache:

Small Queue (memory): intercept occasional queries to avoid unnecessary disk I/O.

Main Queue (local disk): store truly hot segments.

Eviction follows an S3‑FIFO base policy, with a frequency counter that protects frequently accessed segments. Cache size is auto‑managed (≈20‑30 % of cold data size) and cannot be configured manually.

Region alignment – isolation guarantee

Each Region must be either fully IA or fully non‑IA; mixed regions are not allowed because IA and standard data have different cache, compaction, and quota policies. Converting a table to IA triggers a Region split, separating IA data into its own Regions. Switching back merges Regions automatically.

Observability

Cold‑read statistics appear in EXPLAIN ANALYZE output with three new fields: total bytes loaded, load count, and wait time. Slow‑log entries and the Tap monitoring panel also expose IA‑related metrics.

Switching costs

Standard → IA is lightweight: data already resides on S3, so the operation only creates meta files, splits Regions, and gradually GC’s local SST copies. In a test on 1.31 TB of data, the switch completed in under 5 minutes with negligible QPS or latency impact.

IA → Standard requires downloading all data back to local disks. In a 2.09 TB test, the process took about 3 hours 10 minutes, causing a 3.78 % QPS drop and an 18.63 % increase in P99 latency. S3 copies remain for backup and compaction.

Summary

Tiered storage in TiDB is not merely moving “old” data to object storage; it is a comprehensive redesign of the storage stack that balances cost, performance, and isolation. Multi‑tree parallelism, S3 persistence, segment‑level caching, IA Manager, and Region alignment together enable fine‑grained hot‑cold data handling. However, the approach introduces read‑amplification latency for first‑time cold reads and imposes limits on partition types and cache configuration. Proper workload analysis and continuous monitoring of cold‑read ratios, cache hit rates, and object‑storage requests are essential before adopting tiered storage.