Master Loki on Kubernetes: Complete Deployment, Configuration, and Troubleshooting Guide

Why Use Loki

Loki is a lightweight log collection and analysis system that uses promtail to gather logs and stores them in Loki, which can be visualized and queried via Grafana.

Key Advantages

Supported clients: Promtail, Fluentbit, Fluentd, Vector, Logstash, Grafana Agent

Promtail can ingest logs from files, systemd, Windows Event Log, Docker driver, etc.

No required log format – supports JSON, XML, CSV, logfmt, unstructured text

Log queries use the same syntax as metric queries

Dynamic filtering and transformation during queries

Easy metric calculation from logs

Minimal indexing enables dynamic slicing of logs

Cloud‑native support with Prometheus‑style scraping

Log Collection Component Comparison

Loki Architecture

Logs are indexed by timestamp and selected pod labels; the rest is stored as log content.

Example index selector:

{app="loki",namespace="kube-public"}

Log Collection Architecture

Promtail is recommended as a DaemonSet on Kubernetes worker nodes; other collectors can also be used.

Loki Deployment Modes

all (read/write mode) – single node handles both reads and writes.

read/write (read‑write separation) – fronted‑query forwards reads to read nodes; read nodes run

querier

,

ruler

,

frontend

; write nodes run

distributor

,

ingester

.

microservice mode – each role runs as a separate process.

All‑In‑One Deployment

Prepare a ConfigMap with the full Loki configuration (example shown) and apply it to the cluster.

<code>auth_enabled: false
target: all
ballast_bytes: 20480
server:
  grpc_listen_port: 9095
  http_listen_port: 3100
  graceful_shutdown_timeout: 20s
  grpc_listen_address: "0.0.0.0"
  grpc_listen_network: "tcp"
  grpc_server_max_concurrent_streams: 100
  grpc_server_max_recv_msg_size: 4194304
  grpc_server_max_send_msg_size: 4194304
  http_server_idle_timeout: 2m
  http_listen_address: "0.0.0.0"
  http_listen_network: "tcp"
  http_server_read_timeout: 30s
  http_server_write_timeout: 20s
  log_source_ips_enabled: true
  register_instrumentation: true
  log_format: json
  log_level: info
...</code>

Create the ConfigMap:

<code>$ kubectl create configmap --from-file ./loki-all.yaml loki-all</code>

Verify the ConfigMap exists.

Persistent Storage

Use PersistentVolume and PersistentVolumeClaim (e.g., hostPath) to retain logs across container restarts.

<code>apiVersion: v1
kind: PersistentVolume
metadata:
  name: loki
spec:
  hostPath:
    path: /glusterfs/loki
    type: DirectoryOrCreate
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteMany
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: loki
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Gi
  volumeName: loki
</code>

Deploy the StatefulSet and Services (YAML omitted for brevity).

Validate Deployment

When the pod status shows

Running

and the distributor shows

Active

, logs are being ingested.

Troubleshooting

502 Bad Gateway

Check Loki service address (e.g., http://LokiServiceName, http://LokiServiceName.namespace, http://LokiServiceName.namespace:ServicePort) and network/firewall between Grafana and Loki.

Ingester not ready: instance xx:9095 in state JOINING

Wait for the all‑in‑one pod to finish starting.

too many unhealthy instances in the ring

Set

ingester.lifecycler.replication_factor

to 1 for a single‑node deployment.

Data source connected, but no labels received

Ensure Promtail is correctly configured and able to send logs; delete

positions.yaml

if necessary.

Verify Promtail’s target files and configuration.

Source: https://juejin.cn/post/7150469420605767717