How to Build a High‑Performance Monitoring Platform with Prometheus, InfluxDB, and Grafana

The author first outlines the broad set of monitoring points needed for performance testing (OS, application servers, middleware, queues, caches, databases, network, front‑end, load balancers, storage, code, etc.) and then focuses on the most common components to illustrate the monitoring logic.

JMeter + InfluxDB + Grafana data flow

Typical JMeter runs display results in the console, via a plugin, or as generated HTML, but these approaches waste time, are impractical under high concurrency, consume excessive memory for long runs, and make result retrieval cumbersome. The solution is to enable the JMeter Backend Listener, which asynchronously sends metrics to InfluxDB (supported from JMeter 2.13 for Graphite and JMeter 3.3 for InfluxDB).

Key code in InfluxdbBackendListenerClient.java adds metrics for total, successful, and failed transactions and then sends them:

private void addMetrics(String transaction, SamplerMetric metric) {
    // FOR ALL STATUS
    addMetric(transaction, metric.getTotal(), metric.getSentBytes(), metric.getReceivedBytes(), TAG_ALL, metric.getAllMean(), metric.getAllMinTime(),
        metric.getAllMaxTime(), allPercentiles.values(), metric::getAllPercentile);
    // FOR OK STATUS
    addMetric(transaction, metric.getSuccesses(), null, null, TAG_OK, metric.getOkMean(), metric.getOkMinTime(),
        metric.getOkMaxTime(), okPercentiles.values(), metric::getOkPercentile);
    // FOR KO STATUS
    addMetric(transaction, metric.getFailures(), null, null, TAG_KO, metric.getKoMean(), metric.getKoMinTime(),
        metric.getKoMaxTime(), koPercentiles.values(), metric::getKoPercentile);
    metric.getErrors().forEach((error, count) -> addErrorMetric(transaction, error.getResponseCode(),
        error.getResponseMessage(), count));
}

@Override
public void writeAndSendMetrics() {
    if (!copyMetrics.isEmpty()) {
        try {
            if (httpRequest == null) {
                httpRequest = createRequest(url);
            }
            StringBuilder sb = new StringBuilder(copyMetrics.size()*35);
            for (MetricTuple metric : copyMetrics) {
                sb.append(metric.measurement)
                  .append(metric.tag)
                  .append(" ")
                  .append(metric.field)
                  .append(" ")
                  .append(metric.timestamp+"000000")
                  .append("
");
            }
            StringEntity entity = new StringEntity(sb.toString(), StandardCharsets.UTF_8);
            httpRequest.setEntity(entity);
            lastRequest = httpClient.execute(httpRequest, new FutureCallback<HttpResponse>() {
                @Override
                public void completed(final HttpResponse response) {
                    int code = response.getStatusLine().getStatusCode();
                    if (MetricUtils.isSuccessCode(code)) {
                        if (log.isDebugEnabled()) {
                            log.debug("Success, number of metrics written: {}", copyMetrics.size());
                        }
                    } else {
                        log.error("Error writing metrics to influxDB Url: {}, responseCode: {}, responseBody: {}", url, code, getBody(response));
                    }
                }
                @Override
                public void failed(final Exception ex) {
                    log.error("failed to send data to influxDB server : {}", ex.getMessage());
                }
                @Override
                public void cancelled() {
                    log.warn("Request to influxDB server was cancelled");
                }
            });
        } catch (Exception e) {
            // handle exception
        }
    }
}

Metrics are stored in two InfluxDB measurements: events (test lifecycle events) and jmeter (transaction statistics). Grafana is configured with an InfluxDB datasource and the official JMeter dashboard (ID 5496). The author demonstrates a simple JMeter test (10 threads × 10 iterations × 2 HTTP requests = 200 requests) and shows the corresponding Grafana visualizations for TPS and 95th‑percentile response time.

node_exporter + Prometheus + Grafana data flow

For OS‑level metrics, the stack uses node_exporter to expose counters from /proc. The exporter supports many operating systems; the author starts it with ./node_exporter --web.listen-address=:9200 &. Prometheus is downloaded (v2.14.0) and run with a configuration that scrapes the exporter. Grafana imports the official node_exporter dashboard (ID 11074) and queries Prometheus for CPU usage:

avg(irate(node_cpu_seconds_total{instance=~"$node",mode="system"}[30m])) by (instance)
avg(irate(node_cpu_seconds_total{instance=~"$node",mode="user"}[30m])) by (instance)
avg(irate(node_cpu_seconds_total{instance=~"$node",mode="iowait"}[30m])) by (instance)
1 - avg(irate(node_cpu_seconds_total{instance=~"$node",mode="idle"}[30m])) by (instance)

The author compares the Grafana‑displayed CPU counters with the output of top, emphasizing that the values originate from the same kernel counters and therefore have identical meaning.

Conclusion

Understanding the data source and its semantics is crucial for performance analysis; visual tools like Grafana merely present the raw metrics collected by exporters such as JMeter Backend Listener or node_exporter. The article provides a step‑by‑step guide to configure the entire pipeline, from metric generation to storage and real‑time visualization.