Why Loki Beats ELK for Kubernetes Logging: Architecture and Deployment Guide

Background and Motivation

When designing a logging solution for a container cloud, the heavyweight nature of ELK/EFK and the rarely used complex search features of Elasticsearch led to the selection of Grafana’s open‑source Loki system. Loki aims to minimise the cost of switching between metrics and logs, reduce incident response time, and lower storage overhead.

Loki Goals

Minimise the cost of switching between metrics and logs.

Balance query language simplicity with expressive power.

Provide a cost‑effective solution by reducing indexing and storage overhead.

Architecture Overview

Loki’s architecture mirrors Prometheus by using the same label‑based indexing, allowing logs and metrics to be queried with the same selectors. The main components are:

Distributor : First entry point for logs collected by Promtail; batches and compresses data before forwarding.

Ingester : Stateful component that builds compressed chunks (gzip) and flushes them to storage when size or time thresholds are reached; replicates chunks (default replication factor 3) for redundancy.

Querier : Handles read requests by selecting matching chunks based on label selectors, performing distributed grep operations, and merging results with unflushed data from Ingester.

Write Path

Logs are collected by promtail running as a DaemonSet on each node. Promtail attaches Kubernetes metadata as labels and sends logs to the Distributor. The Distributor batches logs, compresses them, and forwards them to an appropriate Ingester based on a hash of the stream’s metadata.

Read Path

The Querier receives a time range and label selector, looks up matching chunks in the index, retrieves them from storage, and streams the results. It also queries the Ingester for the latest unflushed data, enabling near‑real‑time log access.

Scalability

Indexes can be stored in Cassandra, Bigtable, or DynamoDB, while chunks reside in object storage. Both Distributor and Querier are stateless, allowing horizontal scaling. Ingester state is rebalanced when nodes are added or removed, and the underlying Cortex storage has been proven in production for years.

Deployment Steps (OpenShift)

Create a dedicated namespace: oc new-project loki Grant required permissions:

oc adm policy add-scc-to-user anyuid -z default -n loki
oc adm policy add-cluster-role-to-user cluster-admin system:serviceaccount:loki:default

Deploy Loki as a StatefulSet (example statefulset.json):

{
  "apiVersion": "apps/v1",
  "kind": "StatefulSet",
  "metadata": {"name": "loki"},
  "spec": {
    "replicas": 1,
    "serviceName": "loki-headless",
    "template": {
      "metadata": {"labels": {"app": "loki"}},
      "spec": {
        "containers": [{
          "name": "loki",
          "image": "grafana/loki:latest",
          "args": ["-config.file=/etc/loki/local-config.yaml"],
          "ports": [{"containerPort": 3100, "name": "http-metrics"}],
          "livenessProbe": {"httpGet": {"path": "/ready", "port": "http-metrics"}, "initialDelaySeconds": 45},
          "readinessProbe": {"httpGet": {"path": "/ready", "port": "http-metrics"}, "initialDelaySeconds": 45}
        }]
      }
    }
  }
}

Deploy Promtail as a DaemonSet (example daemonset.json) and a ConfigMap containing promtail.yaml (example configmap.json). The ConfigMap defines scrape jobs, relabel rules, and pipeline stages for Kubernetes pod logs.

Create a Service of type NodePort to expose Loki’s HTTP API:

{
  "apiVersion": "v1",
  "kind": "Service",
  "metadata": {"name": "loki"},
  "spec": {
    "type": "NodePort",
    "selector": {"app": "loki"},
    "ports": [{"port": 3100, "targetPort": 3100, "protocol": "TCP", "name": "lokiport"}]
  }
}

LogQL Query Interface

Loki provides an HTTP API (see the official GitHub repository at https://github.com/grafana/loki/blob/master/docs/api.md). A typical query workflow:

Retrieve available label names: curl http://HOST:PORT/api/prom/label Get possible values for a specific label (e.g., namespace):

curl http://HOST:PORT/api/prom/label/namespace/values

Execute a LogQL query using label selectors and optional filters:

curl "http://HOST:PORT/api/prom/query?direction=BACKWARD&limit=1000&query={namespace=\"cicd\"}&start=1567644457221000000&end=1567730857221000000"

Query Parameters

query : LogQL expression, e.g., {name=~"mysql.+"} |= "error".

limit : Maximum number of log entries to return.

start / end : Unix nanosecond timestamps defining the time window (default: one hour ago to now).

direction : FORWARD or BACKWARD, used with limit.

regexp : Optional regex filter applied to the result set.

LogQL Syntax

Selectors are placed inside {} with comma‑separated label matchers. Supported operators: =: Exact match. !=: Not equal. =~: Regex match. !~: Regex not match.

After the selector, filter expressions can be added: |= "text": Keep lines containing the string. != "text": Exclude lines containing the string. |~ "regex": Keep lines matching the regex. !~ "regex": Exclude lines matching the regex.

Multiple filters can be chained, e.g., {job="mysql"} |= "error" != "timeout". The regex syntax follows the RE2 engine (see https://github.com/google/re2/wiki/Syntax).