Step-by-Step Installation and Configuration of Prometheus, Node Exporter, and Grafana on a Kubernetes Cluster

1. Install and configure Prometheus on the master node (192.168.210.85)

Extract the archive and move it to /data/prometheus:

tar -zxvf prometheus-2.23.0.linux-amd64.tar.gz -C /data

mv prometheus-2.23.0.linux-amd64 /data/prometheus

Set ownership: chown prometheus:prometheus -R /data/prometheus Create a systemd service file ( /etc/systemd/system/prometheus.service) with the following content:

[Unit]
Description=Prometheus
After=network.target

[Service]
ExecStart=/data/prometheus/prometheus --config.file=/data/prometheus/prometheus.yml --storage.tsdb.path=/data/prometheus/data
User=prometheus

[Install]
WantedBy=multi-user.target

Enable and start the service:

systemctl enable prometheus

systemctl start prometheus

Verify that Prometheus is listening on port 9090:

netstat -nltp|grep prometheus

2. Install node_exporter on the master and worker nodes

On the master node:

tar -zxvf node_exporter-1.0.1.linux-amd64.tar.gz -C /usr/local/

mv /usr/local/node_exporter-1.0.1.linux-amd64 /usr/local/node_exporter

Run it in the background: nohup /usr/local/node_exporter/node_exporter & Repeat the same steps on k8s-node1 (192.168.210.177) (see the detailed commands in the original text).

Update prometheus.yml to scrape the new targets:

static_configs:
  - targets: ['192.168.210.85:9090']
  - job_name: 'k8s-node1'
  - targets: ['192.168.210.177:9100']
  - job_name: 'k8s-master1'
  - targets: ['192.168.210.85:9100']

Restart Prometheus to apply the changes: systemctl restart prometheus 3. Install and configure Grafana on node2 (192.168.210.195)

Download and extract Grafana:

wget https://dl.grafana.com/oss/release/grafana-7.3.5.linux-amd64.tar.gz

tar -zxvf grafana-7.3.5.linux-amd64.tar.gz

Move it to /usr/local/ and create a symlink:

mv grafana-7.3.5 /usr/local/

ln -s /usr/local/grafana-7.3.5/ /usr/local/grafana

Create a non‑login user for Grafana and set up data directories:

useradd -s /sbin/nologin -M grafana

mkdir -p /data/grafana

Adjust ownership:

chown -R grafana:grafana /usr/local/grafana

chown -R grafana:grafana /data/grafana

Edit /usr/local/grafana/conf/defaults.ini (or a copy) to point to the data directories:

data = /data/grafana/data
logs = /data/grafana/log
plugins = /data/grafana/plugins
provisioning = /data/grafana/conf/provisioning

Create a systemd service file ( /etc/systemd/system/grafana-server.service) with the following content:

[Unit]
Description=Grafana
After=network.target

[Service]
User=grafana
Group=grafana
Type=notify
ExecStart=/usr/local/grafana/bin/grafana-server -homepath /usr/local/grafana
Restart=on-failure

[Install]
WantedBy=multi-user.target

Start and enable the service: systemctl start grafana-server Verify it is listening on port 3000:

netstat -nltp|grep grafana

Access Grafana via a browser at http://192.168.210.195:3000 (default login: admin / admin).

4. Verify basic monitoring of the master and nodes

Open the Prometheus targets page ( http://192.168.210.85:9090/targets) to see the status of the master, node1, and node_exporter instances.

Similarly, view Grafana dashboards to monitor the collected metrics.