Master Loki Logging: Deploy, Configure, and Troubleshoot on Kubernetes

1 Loki

1.1 Introduction

Loki

is a lightweight log collection and analysis system that uses promtail to gather log data and stores it in Loki, which can then be visualized and queried via a grafana datasource.

Loki supports persistent storage back‑ends such as azure, gcs, s3, swift, and local, with s3 and local being the most common. It also integrates with many log shippers like logstash, fluentbit, fluentd, vector, and grafana‑agent.

Advantages

Supports clients such as Promtail, Fluentbit, Fluentd, Vector, Logstash, and Grafana Agent.

Promtail is the preferred agent, capable of pulling logs from local files, systemd, Windows event logs, Docker drivers, etc.

No strict log format required – JSON, XML, CSV, logfmt, or unstructured text are all accepted.

Uses the same query language as Prometheus for log queries.

Allows dynamic filtering and transformation of log lines during queries.

Facilitates easy metric extraction from logs.

Minimal indexing enables slice‑and‑dice queries for new issues.

Cloud‑native ready, leveraging Prometheus‑style scraping.

Component Comparison

Name

Installed Components

Advantages

ELK/EFK

elasticsearch, logstash, kibana, filebeat, kafka/redis

Custom grok parsing, rich visual dashboards

Loki

grafana, loki, promtail

Low resource usage, native Grafana support, fast queries

1.2 Loki Working Principle

1.2.1 Log Parsing Format

Loki parses logs based on an index that includes a timestamp and a pod label (other labels include filename, containers, etc.). The remaining part is the log content. Example query:

{app="loki",namespace="kube-public"}

1.2.2 Log Collection Architecture

The recommended approach is to run promtail as a DaemonSet on each Kubernetes worker node, acting as the log‑collection agent. Alternative shippers can also be used.

1.2.3 Loki Deployment Modes

Loki consists of five micro‑services. Adding a memberlist_config cache enables horizontal scaling. The binary supports the -target flag with values all, read, and write to select the appropriate mode. all (read‑write mode): a single node handles both reads and writes.

read/write

(read‑write split): queries are forwarded to read nodes ( querier, ruler, frontend), while writes go to distributor and ingester nodes.

Micro‑service mode: each component runs as a separate process with its own role.

1.3 Server‑Side Deployment

A Kubernetes cluster is required. The following images are used:

loki: grafana/loki:2.5.0 promtail:

grafana/promtail:2.5.0

1.3.1 All‑In‑One Deployment

1.3.1.1 Create ConfigMap

Save the full configuration (shown below) to loki-all.yaml and create a ConfigMap:

kubectl create configmap --from-file=./loki-all.yaml loki-all

1.3.1.2 Persistent Storage

Define a PersistentVolume and PersistentVolumeClaim for log data:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: loki
spec:
  hostPath:
    path: /glusterfs/loki
    type: DirectoryOrCreate
  capacity:
    storage: 1Gi
  accessModes:
  - ReadWriteMany
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: loki
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 1Gi
  volumeName: loki

1.3.1.3 Deploy StatefulSet

Apply the following StatefulSet and Service definitions:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: loki
  labels:
    app: loki
spec:
  serviceName: "loki"
  replicas: 1
  selector:
    matchLabels:
      app: loki
  template:
    metadata:
      labels:
        app: loki
    spec:
      containers:
      - name: loki
        image: grafana/loki:2.5.0
        args:
        - -config.file=/etc/loki/loki-all.yaml
        ports:
        - containerPort: 3100
          name: http-metrics
        - containerPort: 9095
          name: grpc
        - containerPort: 7946
          name: memberlist-port
        volumeMounts:
        - name: config
          mountPath: /etc/loki
        - name: storage
          mountPath: /data
      volumes:
      - name: config
        configMap:
          name: loki
      - name: storage
        persistentVolumeClaim:
          claimName: loki
---
apiVersion: v1
kind: Service
metadata:
  name: loki
spec:
  ports:
  - name: http
    port: 3100
    targetPort: 3100
  - name: grpc
    port: 9095
    targetPort: 9095
  selector:
    app: loki

1.3.1.4 Verify Deployment

When the StatefulSet shows Running and the distributor reports Active, logs are being ingested correctly. Screenshots in the original article illustrate the status.

1.3.2 Bare‑Metal Deployment

Copy the loki binary to /bin/ and create a systemd unit file /usr/lib/systemd/system/grafana‑loki.service:

[Unit]
Description=Grafana Loki Log Ingester
After=network-online.target

[Service]
ExecStart=/bin/loki --config.file /etc/loki/loki-all.yaml
ExecReload=/bin/kill -s HUP $MAINPID
ExecStop=/bin/kill -s TERM $MAINPID

[Install]
WantedBy=multi-user.target

Reload and start the service with systemctl daemon-reload && systemctl start grafana‑loki.

1.4 Promtail Deployment

Promtail collects logs and pushes them to Loki. Create a configuration file (example below) and store it as a ConfigMap.

server:
  log_level: info
  http_listen_port: 3101
clients:
- url: http://loki:3100/loki/api/v1/push
positions:
  filename: /run/promtail/positions.yaml
scrape_configs:
- job_name: kubernetes-pods
  pipeline_stages:
  - cri: {}
  kubernetes_sd_configs:
  - role: pod
  relabel_configs:
  - source_labels: [__meta_kubernetes_pod_controller_name]
    regex: '([0-9a-z-.]+?)(-[0-9a-f]{8,10})?'
    target_label: __tmp_controller_name
  # ... (additional relabel rules omitted for brevity)

1.4.1 Kubernetes Deployment

1.4.1.1 DaemonSet

Deploy Promtail as a DaemonSet so it runs on every node:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: promtail
  labels:
    app.kubernetes.io/name: promtail
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: promtail
  template:
    metadata:
      labels:
        app.kubernetes.io/name: promtail
    spec:
      serviceAccountName: promtail
      containers:
      - name: promtail
        image: grafana/promtail:2.3.0
        args:
        - -config.file=/etc/promtail/promtail.yaml
        ports:
        - containerPort: 3101
          name: http-metrics
        volumeMounts:
        - name: config
          mountPath: /etc/promtail
        - name: run
          mountPath: /run/promtail
        - name: containers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: pods
          mountPath: /var/log/pods
          readOnly: true
      volumes:
      - name: config
        configMap:
          name: promtail
      - name: run
        hostPath:
          path: /run/promtail
      - name: containers
        hostPath:
          path: /var/lib/docker/containers
      - name: pods
        hostPath:
          path: /var/log/pods

1.4.1.2 Apply DaemonSet

kubectl apply -f promtail.yaml

1.4.2 Bare‑Metal Deployment

Adjust the clients URL to point to the Loki host, place the config under /etc/loki/, and create a systemd unit /usr/lib/systemd/system/loki‑promtail.service:

[Unit]
Description=Grafana Loki Log Ingester
After=network-online.target

[Service]
ExecStart=/bin/promtail --config.file /etc/loki/loki-promtail.yaml
ExecReload=/bin/kill -s HUP $MAINPID
ExecStop=/bin/kill -s TERM $MAINPID

[Install]
WantedBy=multi-user.target

1.5 Adding Loki as a Grafana Data Source

In Grafana go to Settings → Data Sources → Add Data Source → Loki. Use the HTTP URL of the Loki service (e.g., http://loki:3100 when both run in the same namespace). The service name resolves via Kubernetes DNS.

1.6 Other Log Shippers

1.6.1 Logstash Output Plugin

Install the Loki output plugin for Logstash:

bin/logstash-plugin install logstash-output-loki

Example Logstash output configuration:

output {
  loki {
    url => "http://loki:3100/loki/api/v1/push"
    tenant_id => "my-tenant"
    message_field => "message"
    batch_wait => 1
    batch_size => 102400
  }
}

1.7 Helm Installation

For a quick install, use Helm:

# Add the Grafana repo
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
# Install Loki with default settings
helm upgrade --install loki grafana/loki-simple-scalable
# Install into a custom namespace
helm upgrade --install loki --namespace=loki grafana/loki-simple-scalable
# Override values
helm upgrade --install loki grafana/loki-simple-scalable --set "replicaCount=2,resources.limits.cpu=500m"

1.8 Troubleshooting

1.8.1 502 Bad Gateway

Check that the Loki service URL is correct (e.g., http://loki, http://loki.namespace, or http://loki.namespace:3100) and that network/firewall rules allow traffic between Grafana and Loki.

1.8.2 Ingester Not Ready (JOINING)

All‑In‑One mode may need a few seconds to become ready; wait before querying.

1.8.3 Too Many Unhealthy Instances

Set ingester.lifecycler.replication_factor to 1 when running a single instance.

1.8.4 Data Source Connected but No Labels

Ensure Promtail is successfully pushing logs (check its logs).

Verify Promtail’s positions.yaml file is writable and not stale.

Confirm the log file paths in the Promtail config are correct.