Achieving Full Observability for Performance Testing with Prometheus

Observability in Performance Testing

Observability extends traditional monitoring by collecting Metrics, Traces, and Logs. In distributed systems it enables rapid diagnosis of performance problems, and the metric values directly determine whether a load test passes and if the system is ready for production.

Observability Dimensions

Metrics – quantitative measurements such as success rate, throughput, response time, CPU, memory, disk and network usage.

Logs – engine logs for health checks and sampled request/response logs for detailed error analysis.

Traces – distributed tracing that shows the call chain of a request and pinpoints failing APIs and stack traces.

Core Performance‑Testing Metrics

System performance : transaction response time (average RT), processing capacity (HPS, TPS, QPS), concurrent virtual users (VU), error/failure ratio (FR). Typical response‑time targets: Internet < 500 ms, Finance < 1 s, Manufacturing < 5 s.

Resource usage : CPU total ≤ 75 % (sys ≤ 30 %, wait ≤ 5 %), memory swap ≤ 70 %, disk busy ≤ 70 %, network throughput ≤ 70 % of link capacity.

Middleware : GC frequency and duration (including Full GC), heap usage %, active thread count, pending request count, JDBC active connections.

Database : SQL latency (µs), QPS/TPS, cache‑hit rates (≥ 95 %), lock wait count and wait time (µs).

Frontend : first‑paint, onload, full‑load times (ms), page size (KB), request count, DNS lookup, TCP connect, server processing, transfer and wait times (ms).

Stability, Batch, Scalability and Reliability Indicators

Stability : sustain ≥ 80 % of peak capacity for ≥ 8 h (24 h for 24/7 services) with a stable TPS curve and no resource leaks.

Batch processing : high‑volume batch jobs must complete within a short window and must not degrade real‑time transaction performance.

Scalability : performance should increase proportionally with added resources; scaling efficiency ≥ 70 % is considered acceptable.

Reliability : successful failover, node‑switch time, data‑loss during recovery, and cluster behavior when individual nodes fail.

Why Prometheus for Load‑Test Monitoring

Open‑source load generators such as JMeter expose only basic metrics. Prometheus provides a high‑frequency time‑series database that can ingest per‑second metrics from many load‑engine instances, offers native container and Kubernetes monitoring, and scales horizontally.

Comparison with Zabbix

Zabbix stores monitoring data in relational databases, which become a bottleneck under high‑frequency, high‑concurrency ingestion and offers limited container visibility. Prometheus’s pull‑based model and TSDB deliver superior ingestion speed and cloud‑native resource observability.

Integrating Prometheus with JMeter

JMeter can be extended via a custom BackendListener plugin that registers Prometheus counters, updates them after each sampler, and exposes an HTTP endpoint for Prometheus to scrape.

Add a metric registry that creates the required Counter and Gauge objects.

Implement a Prometheus metric updater that maps JMeter sampler results (success count, failure count, latency, etc.) to the counters.

Create a custom BackendListener class that invokes the updater in handleSampleResult() after each sampler execution.

Start an embedded HTTP server (e.g., SimpleHttpServer) that serves the /metrics endpoint; optionally add basic authentication.

Using Alibaba Cloud PTS with Prometheus

Alibaba Cloud Performance Testing Service (PTS) already exports its test‑engine and pressure‑engine metrics to Alibaba Cloud Prometheus. Users can view the metrics, build dashboards, and configure alert rules directly in the Prometheus console without writing custom plugins.

Summary

Observability—collecting metrics, logs, and traces—is essential for reliable performance testing. Prometheus offers a scalable, cloud‑native solution for high‑frequency metric collection and analysis, while PTS provides a ready‑made integration for Alibaba Cloud users, together delivering end‑to‑end visibility of load‑test results.