Scaling Log Indexing: From Full‑Index to Selective Strategies for Million‑QPS Systems

Everyone Starts with Full Indexing

Elasticsearch is the default because its schema‑on‑write model tokenizes every field, builds inverted indexes, and creates columnar doc_values, delivering sub‑second search on any keyword. The write path stores 2–5× the original log size (inverted index, doc_values, _source, plus replicas), which is acceptable at GB scale but becomes costly at larger volumes.

The Five Walls: Costs of Full Indexing at TB Scale

Wall 1 – Storage Amplification : At PB scale the 2–5× blow‑up means the index alone exceeds raw log size, yet 99% of logs are never queried.

Wall 2 – Write Throughput : Tokenization, inverted‑index building, refresh and segment merges are CPU‑intensive; a million logs per second can saturate data‑node CPUs, forcing horizontal scaling just to keep up.

Wall 3 – Mapping Explosion : Dynamic mapping lets every service add arbitrary fields, inflating cluster state from hundreds to tens of thousands of fields and causing type conflicts.

Wall 4 – Cost Mismatch : Real‑world queries are highly predictable (trace_id, service + level, time window), yet full indexing charges uniformly, wasting resources on long‑tail queries.

Wall 5 – Operational Complexity : Shard planning, hotspot mitigation, reindexing, and lifecycle policies require dedicated ops effort to maintain an index that 99% of users never touch.

The Essence of Indexing: Pre‑paying for Future Queries

Indexing trades write‑time cost for low‑latency reads. The key variable is how well you understand future query patterns; the more you know, the more you can shift from blanket pre‑payment to selective payment.

Four Selective Indexing Paths

All four approaches share the premise of indexing only what high‑frequency queries need and scanning the rest.

Path 1 – Reduce Inside Elasticsearch

Keep the engine but thin the index:

Disable dynamic mapping and declare fields explicitly.

Set index: false or enabled: false for unused fields or whole objects.

Use keyword instead of text for fields that only need exact match (e.g., service).

Turn off doc_values for fields that are never aggregated.

Compress _source with a higher compression level.

Result: index size often halves and write throughput improves noticeably, but the underlying Lucene segment model still limits scalability.

Path 2 – Loki’s Philosophy: Index Only Tags

Loki stores raw log chunks in object storage and indexes only low‑cardinality labels such as service, pod, level, cluster, and environment. Queries first filter by these tags and a time window, then perform a distributed grep‑style scan of the relevant chunks.

Benefits: index size drops to <1% of raw data, CPU usage during ingestion is minimal, and storage lives on cheap object storage. Drawback: queries that lack tag or time constraints trigger large scans and become slow.

Path 3 – Column Store + Light‑weight Skip Indexes

ClickHouse treats structured logs as a wide table. Columnar storage yields >10× compression for repetitive log data, and scanning is fast. Adding sparse primary‑key indexes (time + service) narrows the scan range, while skip indexes such as bloom_filter, minmax, and set quickly discard irrelevant data blocks.

Case study: Uber migrated from Elasticsearch to ClickHouse, cutting hardware costs by more than 50% and boosting write throughput several‑fold. Similar results are reported by Bilibili, Kuaishou, VictoriaLogs, and Quickwit.

Path 4 – Time‑Dimension Selective Tiering

Logs are hot, warm, or cold based on age:

Hot layer : full index for the most recent hour(s) to guarantee sub‑second troubleshooting.

Warm layer : reduced index (tags + light skip indexes) for the past few days; queries are slightly slower but acceptable.

Cold layer : no index; raw logs are compressed and stored in S3/OSS, scanned on demand or re‑indexed temporarily.

This mirrors the schema‑on‑write vs. schema‑on‑read split used by Splunk and many commercial log services.

How to Choose – Key Design Points

1. Analyze query logs : quantify the share of exact trace_id lookups, service + level filters, and full‑text searches.

2. Prioritize trace_id : build a precise index or a low‑false‑positive bloom filter for trace_id across all layers.

3. Understand bloom filter trade‑offs : aim for <1% index size while filtering out >90% irrelevant blocks.

4. Ensure structured logs : field governance is a prerequisite for any selective strategy; without stable schemas, field‑level indexing is impossible.

Typical practical combo: structured logs as foundation, field indexes on trace_id, service, level, and time window; storage on ClickHouse or Loki; hot layer retains full index for recent data; warm/cold layers rely on tag or no index.

From Pre‑pay to Pay‑as‑You‑Go

The initial cost‑review meeting revealed that the problem is not log volume but an outdated GB‑era indexing strategy. At TB‑ and PB‑scale, paying uniformly for every possible query inflates costs to astronomical levels.

In summary, moving from full‑index to selective indexing means paying only for high‑frequency, predictable queries and scanning the long tail on demand, a shift essential for systems handling millions of QPS and hundreds of terabytes of logs per day.