How to Achieve Full Observability for Performance Testing with Prometheus

What is observability in performance testing?

Observability consists of three dimensions—Metrics, Traces, and Logs—and helps quickly locate and diagnose issues in complex distributed systems.

Key metrics for performance testing

Metrics (monitoring indicators) : system performance metrics such as request success rate, throughput (TPS), and response time; resource metrics like CPU, memory, disk I/O, network throughput, database connections, and cache hit rate.

Logs : pressure‑engine logs to monitor engine health and script errors; sampling logs that record API request and response details for deeper troubleshooting.

Traces : distributed tracing to follow request call chains, pinpoint failing APIs, and identify the exact stack trace causing performance bottlenecks.

Why use Prometheus for performance testing monitoring?

Compared with Zabbix, which stores data in relational databases and struggles with high‑frequency, large‑scale data, Prometheus uses a time‑series database that offers high write/read performance, native support for container and Kubernetes environments, and efficient aggregation of massive per‑second metrics.

Extending JMeter with a Prometheus exporter

Add a metric registry to collect custom counters.

Implement a Prometheus metric updater that pushes the collected values.

Create a custom BackendListener that updates metrics after each sampler execution.

Expose an HTTP endpoint (optionally with authentication) for Prometheus to scrape.

Architecture diagram:

Using Alibaba Cloud PTS with Prometheus

PTS can export its performance metrics to Prometheus without custom plugins. The integration steps are:

Enable the Prometheus switch in PTS advanced settings.

After the test starts, copy the generated Prometheus configuration from the “Monitoring Export” section.

Paste the configuration into your own Prometheus instance and reload it.

Importing a Grafana dashboard

The official Grafana dashboard template (ID 15981) visualizes the exported metrics. Import steps:

In Grafana, go to Dashboard → Import .

Enter the dashboard ID 15981 and click Load .

Select your Prometheus data source and click Import .

Dashboard preview:

The dashboard displays global request success rate, TPS, 99/95/90‑percentile response times, error counts by status code, and JVM garbage‑collection metrics of the load generator.

Summary

Observability is indispensable for performance testing; Prometheus offers a scalable, cloud‑native solution for collecting metrics, logs, and traces. By customizing JMeter or leveraging PTS, you can expose the necessary data, and Grafana dashboards provide a clear visual overview for rapid issue detection and capacity planning.