Boosting etcd Write Performance on AWS: From gp2 Limits to GP3 and Beyond

Testing Write Performance

First, the team ran etcdctl check perf with different --load levels (s, m, l, xl). The small load passed, while the large load failed, indicating the cluster can sustain about 6.6 K writes per second, placing it between medium and large clusters.

iostat showed that the write‑ahead‑log volume (nvme1n1) was at 100 % I/O utilization, causing etcd threads to wait for I/O.

fio Latency Measurements

Running fio with --fdatasync=1 yielded a 99th‑percentile fdatasync latency of 2671 µs (2.7 ms), well below the etcd recommendation of 10 ms. The reported IOPS was 709, close to the gp2 provisioned 900 IOPS.

Upgrading to GP3

After switching the volumes to GP3 (minimum 3000 IOPS), fio reported 1105 IOPS, but the bottleneck remained the EBS volume.

Pushing IOPS to Instance Limits

By increasing the instance’s maximum IOPS to ~6000, fio showed an IOPS of about 1129, a modest 30 % gain over GP3, indicating diminishing returns.

Why IOPS Aren’t Fully Utilized

Operating systems cache writes; data remains in cache until flushed. etcd performs an fdatasync after each transaction, which blocks until the write is persisted, adding latency.

Transaction‑sensitive applications need low I/O latency and benefit from SSDs, while throughput‑sensitive workloads can tolerate higher latency with HDDs.

The AWS documentation notes that continuously increasing volume IOPS raises latency.

Volume Performance Limits

A table of maximum performance for different volume types shows that etcd’s synchronous write speed is constrained both by its implementation and the underlying storage.

Appendix: fio Commands for etcd Storage Testing

etcd exposes the wal_fsync_duration_seconds metric; the 99th percentile should be under 10 ms. The following fio command reproduces the test:

fio --rw=write --ioengine=sync --fdatasync=1 --directory=test-data --size=22m --bs=2300 --name=mytest

Key points: adjust --size and --bs for the scenario, ensure fio version ≥ 3.5, and understand that other writes may affect the observed latency.

etcd uses a write‑ahead log (WAL); each write must be persisted with fdatasync to guarantee durability.