Benchmarking Citus and PostgreSQL with HammerDB: Achieving 2 Million NOPM

Why automation matters for PostgreSQL benchmarks

Running performance benchmarks repeatedly is essential when tuning database configurations, but manual setup is time‑consuming; automating the process with scripts and tools reduces effort while ensuring repeatability.

Choosing the right benchmark for your workload

Different workloads (OLTP, OLAP, HTAP) require different benchmark suites. The article outlines the characteristics of each workload type and maps them to standard benchmarks such as YCSB, TPC‑C, TPC‑H, and HammerDB TPROC‑C/H.

Benchmark specifications and the NOPM metric

HammerDB reports results in NOPM (new‑orders‑per‑minute) instead of TPS, aligning with the tpmC metric from the official TPC‑C specification, enabling comparable numbers across databases.

OLTP workload details

OLTP workloads involve many short transactions (insert, update, delete) that affect a single row and require sub‑100 ms response times; common benchmarks include YCSB, TPC‑C, and HammerDB TPROC‑C.

OLAP workload details

OLAP workloads run large analytical queries that read most of the data, use aggregates, joins, CTEs, subqueries, and window functions, and tolerate longer runtimes (seconds to minutes). Benchmarks include TPC‑H, TPC‑DS, and HammerDB TPROC‑H.

HTAP workload challenges

HTAP combines OLTP and OLAP, leading to a trade‑off: increasing TPS improves transactional throughput but can degrade analytical query latency because CPU/disk resources are shared and data volume grows faster.

Risks of comparing online benchmark results

When comparing published results, consider differences in production features (HA, TLS), dataset size relative to RAM, hardware cost, and whether the benchmark implementation faithfully follows the specification.

HammerDB TPROC‑C for OLTP

HammerDB provides open‑source implementations of TPROC‑C (based on TPC‑C) and TPROC‑H (based on TPC‑H), allowing direct comparison of different databases.

Open‑source benchmark automation tool

The author contributed a GitHub repository ( citus‑benchmark) that automates HammerDB runs on Azure using ARM/Bicep templates, tmux, and cloud‑init. The tool provisions a Citus cluster, a driver VM, and runs benchmarks in the background.

Citus configuration tips

Distribute PostgreSQL tables manually before loading data.

Choose the correct distribution column based on benchmark queries.

Run VACUUM ANALYZE on all tables after data generation.

Set shard_count to a multiple of the worker count (default 48 works well).

Running the benchmark with citus‑benchmark

Execute a single script (see the repository’s README) which configures four Citus clusters and four 64‑vCore driver VMs, then streams results to results.csv. The cluster identifier (e.g., c4+2w8) encodes coordinator and worker vCore counts.

Achieving 2 million NOPM on Azure

A large Citus cluster (64‑core coordinator, 20 workers with 32 cores each, total 704 cores) and increased HammerDB connections (5000) yielded up to 2 million NOPM. Raising the maximum user connections on the Azure Hyperscale (Citus) service from the default 1000 to at least 5000 required a support ticket.

Conclusion

By selecting the appropriate benchmark, automating deployment with ARM/Bicep, tmux, and cloud‑init, and applying Citus‑specific configuration best practices, developers can reliably benchmark PostgreSQL workloads on Azure and scale performance to multi‑million NOPM levels.