Mastering Loki: Lightweight Log Aggregation for Kubernetes with Helm

Loki is an open‑source, horizontally scalable, highly available, multi‑tenant log aggregation system from Grafana Labs that follows the same label‑based approach as Prometheus, avoiding full‑text indexing to stay lightweight and cost‑effective.

Key Features

No full‑text indexing; only metadata labels are indexed.

Uses Prometheus‑style labels for indexing and grouping log streams.

Optimized for Kubernetes pod logs; pod labels are automatically indexed.

Native integration with Grafana.

Components

loki

– the main server that stores logs and handles queries. promtail – agent that collects logs and forwards them to Loki (also compatible with Fluentd, etc.).

Grafana – UI for visualising logs.

Installation

Deploy the Loki stack with Helm:

helm repo add loki https://grafana.github.io/loki/charts
helm repo update
helm upgrade --install loki loki/loki-stack

Or download and install manually:

helm pull loki/loki-stack
tar xf loki-stack-2.1.2.tgz
helm install loki loki-stack/

Architecture

Configure Nginx‑Ingress Logging

Update the ConfigMap to enable JSON log format and increase body size, then modify the ingress‑nginx Deployment to mount the log directory and set appropriate permissions.

# Example ConfigMap snippet
apiVersion: v1
kind: ConfigMap
metadata:
  name: ingress-nginx-controller
data:
  access-log-path: /var/log/nginx/access.log
  log-format-escape-json: "true"
  log-format-upstream: '{"timestamp":"$time_iso8601","requestID":"$req_id",...}'

After applying the changes, logs are written to /var/log/nginx on the host.

Configure Promtail

Enable the promtail module in values.yaml and set the node selector, volume mounts, and scrapeConfigs to collect the Nginx‑Ingress logs.

promtail:
  enabled: true
  nodeSelector:
    kubernetes.io/hostname: "k8s-master-134"
  volumeMounts:
    - name: nginx
      mountPath: /var/log/nginx
      readOnly: true
  volumes:
    - name: nginx
      hostPath:
        path: /var/log/nginx
  scrapeConfigs:
    - job_name: kubernetes-nginx-ingress
      static_configs:
        - targets: [localhost]
          labels:
            job: nginx_access_log
            app: ingress-nginx
            __path__: /var/log/nginx/*.log

Upgrade the Loki chart to apply the new promtail configuration:

helm upgrade loki loki-stack/

Grafana Integration

Add Loki as a data source in Grafana (Grafana version ≥ 6.6) and import a dashboard (e.g., ID 12559) to visualise Nginx‑Ingress logs.

After configuring the datasource and dashboard, you can explore logs, filter by labels such as filename, and verify that Nginx‑Ingress logs are being collected correctly.