Quickwit Benchmark: Indexing 23 TB of GitHub Events and Evaluating Search Performance & Cost

Introduction

This article presents a detailed benchmark of Quickwit, a cloud‑native search engine, by indexing the GitHub Archive dataset (approximately 23 TB and 6.1 billion documents) and measuring both search performance and associated costs.

Choosing an Adversarial Dataset

The GitHub Archive was selected because of its large scale (23 TB, 6 billion documents), complex JSON structure with thousands of fields, historical relevance, and easy relevance for developers.

Benchmark Setup

A representative document from the dataset is shown below:

{
  "id": "26163418660",
  "type": "IssuesEvent",
  "actor": { "id": 121737278, "login": "LaymooDR", ... },
  "repo": { "id": 383940088, "name": "ShadowMario/FNF-PsychEngine", ... },
  "payload": { "action": "opened", "issue": { "title": "Chart Editor glitched out..." } },
  "created_at": "2023-01-01T00:00:00Z"
}

The indexing configuration uses inverted indexes for all fields, columnar storage for all but five large‑text fields, and row storage for the full document. The configuration is provided in a public gist.

Deployment and Infrastructure

Deployed Quickwit on Amazon EKS using the official Helm chart.

Stored shards and metadata in AWS S3.

Allocated up to 24 c5.xlarge indexer nodes (4 CPU, 8 GB RAM, $0.1540/h) and up to 32 c5n.2xlarge searcher nodes (8 CPU, 21 GB RAM, $0.4320/h).

Ingested the dataset via an experimental HTTP source that pulls files directly from https://data.gharchive.org/....

Collected metrics with Prometheus and visualized them in Grafana.

Benchmark Results – Indexing Performance

Indexing throughput averaged 27 MB/s per active indexer, resulting in 61 billion indexed documents, an index size of 8.4 TB (compression ratio 2.75), 715 shards, and about 8.5 million documents per shard.

Scalability was linear up to 24 indexers, with the first indexer finishing in 7 hours and the last in 9 hours. Resource utilization was near the limit of each c5.xlarge instance.

Adjusted the merge strategy to limit each shard to a single merge, preventing CPU‑bound merge backlog. Increased the default heap size from 2 GB to 3 GB to allow larger shards to be committed.

Indexing Cost

Instance cost dominates indexing cost. S3 PUT requests ($0.005 per 1 000) are negligible.

Total cost = $0.1540/h * 23 TB / 27 MB/s / 3600 s/h = $36.4
Cost per TB = $1.58/TB

Using 24 indexers and an average throughput of 22 MB/s yields $44.7 total cost, or $1.94 per TB.

Search Performance

Four representative queries were executed with searcher counts ranging from 1 to 32:

Q1 – Term query on actor.login:Cyan4973 (≈22 k hits).

Q2 – Daily histogram of events (full‑dataset scan).

Q3 – Histogram for events of user Cyan4973 (aggregation over 22 k docs).

Q4 – Term aggregation across three text fields to find repositories mentioning “Quickwit”, sorted by stars.

Without caching, 90th‑percentile latency plateaued between 1.4 s and 2.6 s, limited by S3 latency. IO dominated even the CPU‑intensive Q2.

Cache Mechanisms

Split Footer Cache (Hotcache) – avoids fetching hot cache for each shard; configurable via split_footer_cache_capacity (default 500 M).

Fast Field Cache – stores columnar data per shard; configurable via fast_field_cache_capacity (default 1 G).

Partial Request Cache – caches query results per shard (not used in this benchmark).

Enabling hot cache reduced Q1 90th‑percentile latency to below 1 s (800 ms on a single searcher) and lowered overall latency by 2–4 s for single‑searcher runs, with diminishing returns as searcher count increased.

Enabling both hot cache and fast‑field cache brought Q3 latency to ~480 ms, slightly faster than Q1.

Latency stabilization varies with S3 latency; higher searcher counts sometimes show marginally higher latency due to IO contention.

Search Cost

Estimating search cost is more complex because GET request volume is no longer negligible. Approximate costs:

Simple search queries (Q1, Q3) require ~2 000 GET requests for 23 TB.

Analytical queries (Q2) require ~3 000 GET requests.

CPU time for simple queries is under 1 s; the analytical Q2 consumes about 30 s of CPU on the full dataset.

Conclusion

The benchmark demonstrates that Quickwit can index massive datasets with modest resources and achieve sub‑second search latency on 23 TB of data using a single searcher with hot cache. Indexing cost is roughly $2 per TB, and storage cost is about $8.4 per TB per month (compression ratio 2.75). Future work includes adding Lambda support and local‑disk shard caching to further reduce GET request costs.