Setting Up InfluxDB and Grafana for Flink Metrics Monitoring

Introduction

The article describes how to use InfluxDB, a Go‑based time‑series database, together with Grafana to collect and display Flink job metrics, providing a complete step‑by‑step configuration and deployment process.

Hardware Parameters

CPU: Intel E5 v4 12C/24T

Memory: 96 GB

Disk: 2 × 500 GB SSD

Network: 10 Gbps

OS: CentOS 7.5 64‑bit

InfluxDB 1.8

Grafana 6.7.4

Install and Configure InfluxDB

Download the RPM package and install it with yum localinstall to resolve dependencies automatically.

wget https://dl.influxdata.com/influxdb/releases/influxdb-1.8.0.x86_64.rpm
yum -y localinstall influxdb-1.8.0.x86_64.rpm

The main configuration file is /etc/influxdb/influxdb.conf. Important sections include:

[meta]
  dir = "/data1/influxdb/meta"
[data]
  dir = "/data2/influxdb/data"
  wal-dir = "/data1/influxdb/wal"
[coordinator]
  write-timeout = "20s"
  max-concurrent-queries = 0
  query-timeout = "60s"
  log-queries-after = "30s"
[retention]
  enabled = true
  check-interval = "60m"
[http]
  enabled = true
  bind-address = ":8086"
  auth-enabled = false
  log-enabled = false

Start InfluxDB and Create Database

Run InfluxDB in the background and redirect logs:

nohup influxd -config /etc/influxdb/influxdb.conf > /var/log/influxdb/influxd.log 2>&1 &

Enter the InfluxDB shell and create a database for Flink metrics:

~ influx
Connected to http://localhost:8086 version 1.8.0
InfluxDB shell version: 1.8.0
> CREATE DATABASE flink_metrics;
> SHOW DATABASES;

Create a one‑week retention policy and set it as default:

> CREATE RETENTION POLICY "one_week" ON "flink_metrics" DURATION 168h REPLICATION 1 DEFAULT;
> SHOW RETENTION POLICIES ON "flink_metrics";

Configure Flink Metrics Reporter

Copy flink-metrics-influxdb-<version>.jar to $FLINK_HOME/lib and add the following entries to flink-conf.yaml:

metrics.reporter.influxdb.class: org.apache.flink.metrics.influxdb.InfluxdbReporter
metrics.reporter.influxdb.host: bd-flink-mon-001
metrics.reporter.influxdb.port: 8086
metrics.reporter.influxdb.db: flink_metrics

After launching a Flink job on YARN, metrics appear as measurements in the flink_metrics database.

Install and Start Grafana

wget https://dl.grafana.com/oss/release/grafana-6.7.4-1.x86_64.rpm
yum -y localinstall grafana-6.7.4-1.x86_64.rpm
service grafana-server start

Access Grafana at http://<your-host>:3000.

Add InfluxDB Data Source

In Grafana, go to Configuration → Data Sources → Add data source and select InfluxDB, filling in the connection details (URL http://localhost:8086, database flink_metrics, etc.).

Create Flink Metrics Dashboard

Create a new dashboard, add variables for job name and TaskManager ID, and build panels (e.g., line charts) to visualize metrics such as CPU load, JVM memory, and operator heartbeat rates.

Tip: To avoid metric collisions between different runs of the same job, include additional identifiers (e.g., Maven profile or timestamp) in the Flink job name.

Additional Code Example

public static String getJobName(Class<?> clazz, Properties props) {
  return StringUtils.join(
    Arrays.asList(
      clazz.getCanonicalName(),
      new LocalDateTime().toString("yyyyMMddHHmmss"),
      props.getProperty("profile.id")
    ),
    '_'
  );
}

The article ends with a call‑to‑action encouraging readers to like, bookmark, and share the post.