How Prometheus Memory Usage Was Halved: Insights from Bryan Boreham’s KubeCon Talk

Grafana Labs distinguished engineer Bryan Boreham presented at KubeCon+CloudNativeCon a detailed account of how he reduced Prometheus memory usage.

Prometheus memory consumption is a common cause of system crashes; Boreham’s talk titled Prometheus How to Halve Memory Usage described his research, focusing on the heavy memory cost of labels and the methods he used to cut that cost.

According to the Prometheus documentation, labels are used to distinguish the characteristics of the thing being measured.

api_http_requests_total – distinguishes request type: operation=create|update|delete api_request_duration_seconds – distinguishes request phase: stage=extract|transform|load Boreham contributed 30 pull requests, modifying over 2,500 lines of code, and his work helped the latest Prometheus version use roughly half the memory of previous releases.

“It’s been a long road, but the result is very satisfying. Hundreds of thousands of Prometheus servers are running; by lowering memory requirements we reduce both cost and carbon footprint.” – Boreham, after KubeCon+CloudNativeCon, speaking to The New Stack

Go’s Memory Profiler

The Go runtime includes a built‑in profiler that can provide a visual flame‑graph view of Prometheus memory and CPU usage. The width of each block represents the proportion of memory used, with the top of the chart showing 100% (6.7 GB in the example).

The “sawtooth” effect appears in the chart: garbage accumulates over time, is then collected, causing a sharp drop, then accumulates again.

The Go memory profiler reports memory usage at the last garbage collection, so you never see the garbage itself in the graph. Reducing memory consumption starts with asking “what makes it so large?” In Prometheus, roughly one‑third of memory was taken by labels (about 31%).

Prometheus Label Issue

Each series in Prometheus is uniquely identified by a set of name/value pairs. The data structure includes a 24‑byte header for the slice of all labels and a 16‑byte header for each string, meaning the pointers dominate the memory footprint.

Using PR 10991, Boreham packed all strings into a single buffer and encoded their lengths, dramatically reducing overhead. (Images omitted for brevity.)

In Prometheus 2.74.2, the previous version would crash at 17 GB memory usage, but after the changes (version 2.47.2) memory consumption dropped to 13.1 GB with no incidents.

Additional features such as sample processing and native histograms added in 2.47 did not exhaust memory, and a bug fixed in 2.39 (the “transaction isolation ring”) further reduced memory to 10 GB.

Boreham continued to investigate the Go analyzer to target the biggest memory consumers, eventually achieving an overall reduction to about 8 GB – roughly a 50 % cut.

The Go runtime’s GOGC parameter defaults to 100; adjusting it can control the sawtooth growth, which is especially useful for deployments with 100 GB of Prometheus memory.

Today Prometheus runs comfortably under 8 GB, achieving the 50 % reduction goal Boreham set.

“It’s a labor of love – making programs smaller and faster is an obsession for me, and doing it for such a popular open‑source project is fantastic. At Grafana Labs, open source is in our DNA.” – Boreham to The New Stack

You can watch the full KubeCon presentation here: https://youtu.be/29yKJ1312AM