Lightweight Kubernetes Log Collection with Loki and Promtail: Deployment Guide

1. Why Use Loki

Loki is a lightweight log aggregation system developed by Grafana Labs that stores logs alongside Prometheus metrics, supports multiple clients (Promtail, Fluentbit, Fluentd, Vector, Logstash, Grafana Agent), and offers flexible storage backends (Azure, GCS, S3, Swift, local).

Key Advantages

Supports various agents for log collection.

No strict log format requirements.

Query logs using the same syntax as Prometheus.

Cloud‑native, integrates with Prometheus.

Efficient indexing and dynamic slicing.

2. Loki Architecture and Modes

Loki consists of several micro‑services (distributor, ingester, querier, ruler, compactor, etc.) that can be deployed in different modes: All‑In‑One, read/write separation, and pure micro‑service mode.

All‑In‑One Deployment Example

auth_enabled: false
target: all
server:
  http_listen_port: 3100
  grpc_listen_port: 9095
# ... (other component configurations) ...

Save the above as loki-all.yaml and create a ConfigMap:

kubectl create configmap --from-file ./loki-all.yaml loki-all

Persistent Storage

Define a PersistentVolume and PersistentVolumeClaim to retain logs across pod restarts:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: loki
spec:
  hostPath:
    path: /glusterfs/loki
    type: DirectoryOrCreate
  capacity:
    storage: 1Gi
  accessModes:
  - ReadWriteMany
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: loki
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 1Gi
  volumeName: loki

StatefulSet Deployment

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: loki
spec:
  replicas: 1
  selector:
    matchLabels:
      app: loki
  template:
    metadata:
      labels:
        app: loki
    spec:
      containers:
      - name: loki
        image: grafana/loki:2.5.0
        args:
        - -config.file=/etc/loki/loki-all.yaml
        ports:
        - containerPort: 3100
          name: http-metrics
        - containerPort: 9095
          name: grpc
        volumeMounts:
        - name: config
          mountPath: /etc/loki
        - name: storage
          mountPath: /data
      volumes:
      - name: config
        configMap:
          name: loki
      - name: storage
        persistentVolumeClaim:
          claimName: loki

3. Promtail Deployment

Promtail runs as a DaemonSet to ship pod logs to Loki. Example configuration ( promtail.yaml ) includes server settings, client endpoint, positions file, and Kubernetes service discovery with relabeling.

server:
  http_listen_port: 3101
  log_level: info
clients:
- url: http://loki:3100/loki/api/v1/push
positions:
  filename: /run/promtail/positions.yaml
scrape_configs:
- job_name: kubernetes-pods
  kubernetes_sd_configs:
  - role: pod
  relabel_configs:
  - source_labels: [__meta_kubernetes_pod_node_name]
    target_label: node_name
  # ... additional relabel rules ...

Create a ConfigMap with this file and a DaemonSet manifest that mounts the config, /run/promtail, container logs, and pod log directories.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: promtail
spec:
  selector:
    matchLabels:
      app: promtail
  template:
    metadata:
      labels:
        app: promtail
    spec:
      containers:
      - name: promtail
        image: grafana/promtail:2.3.0
        args:
        - -config.file=/etc/promtail/promtail.yaml
        volumeMounts:
        - name: config
          mountPath: /etc/promtail
        - name: run
          mountPath: /run/promtail
        - name: containers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: pods
          mountPath: /var/log/pods
          readOnly: true
      volumes:
      - name: config
        configMap:
          name: promtail
      - name: run
        hostPath:
          path: /run/promtail
      - name: containers
        hostPath:
          path: /var/lib/docker/containers
      - name: pods
        hostPath:
          path: /var/log/pods

4. Grafana Integration

Add Loki as a data source in Grafana (Settings → Data Sources → Add Data Source → Loki) using the service URL http://loki:3100 . After saving, you can query logs using LogQL directly in Grafana dashboards.

5. Alternative Log Shippers

Logstash can forward logs to Loki via the logstash-output-loki plugin or via an HTTP output block. Example Logstash output configuration:

output {
  loki {
    url => "http://loki:3100/loki/api/v1/push"
    batch_wait => 1
    batch_size => 102400
  }
}

6. Helm Installation

For quick deployment, use the official Helm chart:

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
helm upgrade --install loki grafana/loki-simple-scalable --namespace=loki

7. Troubleshooting

502 Bad Gateway – verify Loki service URL and network connectivity.

Ingester not ready – allow time for All‑In‑One startup.

Replication factor errors – set ingester.lifecycler.replication_factor to 1 for single‑node setups.

No labels received – ensure Promtail is running and positions file is cleared if Loki restarts.

Refer to the official Loki documentation for detailed explanations of each component and advanced configuration options.