Loki Log System Overview, Architecture, and Deployment Guide

When designing a logging solution for a container cloud, the author found traditional ELK/EFK stacks too heavyweight and chose Grafana's open‑source Loki system for its low resource usage and tight integration with Prometheus.

Background and Motivation Loki aims to minimize the cost of switching between metrics and logs, reduce response time to incidents, and avoid the complexity of full‑text search indexing that many users never need.

Architecture Loki uses the same label‑based indexing as Prometheus, allowing queries across logs and metrics with minimal overhead. The system consists of three main components:

Distributor : receives log streams from promtail agents, batches and compresses them before passing to ingesters.

Ingester : builds compressed chunks, stores them locally, and flushes full chunks to the configured object store (e.g., S3, GCS, Cassandra, Bigtable).

Querier : selects chunks based on time range and label selectors, performs distributed greps, and merges results with any in‑memory data from ingesters.

The architecture is stateless for Distributor and Querier, while Ingester is stateful but can rebalance chunks when nodes are added or removed, providing high availability and fault tolerance.

Scalability Loki’s index can be stored in scalable back‑ends such as Cassandra, Bigtable, or DynamoDB, and log chunks can reside in any object storage. Replication (default three copies) ensures durability.

Deployment Steps

Create a namespace for Loki: oc new-project loki Set necessary permissions:

oc adm policy add-scc-to-user anyuid -z default -n loki<br/>oc adm policy add-cluster-role-to-user cluster-admin system:serviceaccount:loki:default

Deploy Loki using a StatefulSet (example statefulset.json):

{
  "apiVersion": "apps/v1",
  "kind": "StatefulSet",
  "metadata": {"name": "loki"},
  "spec": {
    "replicas": 1,
    "selector": {"matchLabels": {"app": "loki"}},
    "serviceName": "loki-headless",
    "template": {
      "metadata": {"labels": {"app": "loki"}},
      "spec": {
        "containers": [{
          "name": "loki",
          "image": "grafana/loki:latest",
          "args": ["-config.file=/etc/loki/local-config.yaml"],
          "ports": [{"containerPort": 3100, "name": "http-metrics"}],
          "livenessProbe": {"httpGet": {"path": "/ready", "port": "http-metrics"}, "initialDelaySeconds": 45},
          "readinessProbe": {"httpGet": {"path": "/ready", "port": "http-metrics"}, "initialDelaySeconds": 45},
          "volumeMounts": [{"name": "storage", "mountPath": "/tmp/loki"}]
        }],
        "volumes": [{"name": "storage", "emptyDir": {}}]
      }
    }
  }
}

Deploy Promtail as a DaemonSet (example daemonset.json) to collect logs from each node:

{
  "apiVersion": "apps/v1",
  "kind": "DaemonSet",
  "metadata": {"name": "promtail", "labels": {"app": "promtail"}},
  "spec": {
    "selector": {"matchLabels": {"app": "promtail"}},
    "template": {
      "metadata": {"labels": {"app": "promtail"}},
      "spec": {
        "containers": [{
          "name": "promtail",
          "image": "grafana/promtail:v0.3.0",
          "args": ["-config.file=/etc/promtail/promtail.yaml", "-client.url=http://loki.loki.svc:3100/api/prom/push"],
          "volumeMounts": [
            {"name": "config", "mountPath": "/etc/promtail"},
            {"name": "run", "mountPath": "/run/promtail"},
            {"name": "docker", "mountPath": "/var/lib/docker/containers", "readOnly": true},
            {"name": "pods", "mountPath": "/var/log/pods", "readOnly": true}
          ]
        }],
        "volumes": [
          {"name": "config", "configMap": {"name": "lame-zorse-promtail"}},
          {"name": "run", "hostPath": {"path": "/run/promtail"}},
          {"name": "docker", "hostPath": {"path": "/var/lib/docker/containers"}},
          {"name": "pods", "hostPath": {"path": "/var/log/pods"}}
        ]
      }
    }
  }
}

Create a Service to expose Loki:

{
  "apiVersion": "v1",
  "kind": "Service",
  "metadata": {"name": "loki", "namespace": "loki"},
  "spec": {
    "type": "NodePort",
    "selector": {"app": "loki"},
    "ports": [{"name": "lokiport", "port": 3100, "targetPort": 3100}]
  }
}

After deployment, Loki provides an HTTP API for querying logs using LogQL. Example queries retrieve label values, filter by namespace, and perform regex searches, e.g.,

http://<host>:<port>/api/prom/query?query={namespace="cicd"}&limit=1000&direction=BACKWARD

.