Why Performance Isn't Enough: Rethinking Database Benchmarks

1. The Performance Cult

Database vendors obsess over making faster engines, but users care about the total time to get answers, not just server‑side latency; benchmarks that rank databases by raw speed often ignore real‑world workflow and data‑access patterns.

Choosing a database based only on benchmark scores is a poor strategy; factors such as usability, ecosystem, update cadence, and workflow integration should dominate the decision.

2. Benchmark Wars End

In 2019 GigaOm benchmarked major cloud data warehouses (Azure, Redshift, Snowflake, BigQuery) using TPC‑H and TPC‑DS, finding Azure fastest, while Snowflake and BigQuery lagged.

Despite the results, customers still preferred Snowflake and BigQuery, showing a disconnect between benchmark rankings and market adoption.

Benchmarks are useful but can be misleading if they test the wrong workloads or ignore real‑world data sizes and query patterns.

3. What Does “Fast” Really Mean?

Engineers often focus on server‑side response time, yet users perceive performance as the time from asking a question to receiving an answer, which includes client‑side drivers, data transfer, and result rendering.

In BigQuery, JDBC driver inefficiencies added seconds to queries, dwarfing the engine’s own speed improvements.

Ignoring client‑side latency leads to optimizations that do not improve the user’s actual experience.

4. Performance Is Subjective

Performance must be judged from the user’s perspective; a database that feels fast for a specific workload may be slower for another.

Benchmarks like Clickbench, which only scan single tables, can misrepresent performance for more complex analytical workloads.

Vendor‑driven benchmarks often highlight strengths while hiding weaknesses, and real‑world user experience may differ dramatically.

5. Future Changes

Database capabilities evolve rapidly; a system that is slower today may overtake competitors in a year, so choosing a platform with a strong roadmap is crucial.

6. No Magic

Given enough time, performance gaps between well‑maintained databases tend to narrow as each adopts similar optimizations.

Any performance advantage is usually due to specific engineering tricks that can be replicated elsewhere.

7. The Real Bottleneck Is Between the Chair and the Keyboard

User‑centric metrics—how quickly a question can be asked and answered—are more important than raw query execution time.

Ease of query writing, data format handling (e.g., CSV parsing), and result delivery have a huge impact on productivity.

8. Conclusions

Successful database companies win by making work easier, not by being the fastest in benchmarks; performance alone is insufficient for market success.

No magic; performance converges over time unless architectural differences exist.

Engines evolve at different speeds; the fastest movers win.

Beware of vendors that over‑emphasize performance.

There is no single performance metric; “fast” can hurt workloads.

The key metric is time from idea to answer, not query‑to‑result latency.

Choose databases based on a holistic view that includes usability, ecosystem, and long‑term evolution, not just raw speed.