Why Loki Beats ELK for Kubernetes Logging: Architecture, Deployment, and Query Basics

Background and Motivation

When designing a log solution for a container cloud, the author found ELK/EFK too heavy and chose Grafana Loki because many Elasticsearch features were unnecessary.

In a Kubernetes environment, pods write logs to stdout/stderr. Alerts from Prometheus indicate a problem, but without a log system the operator must manually fetch pod logs, which is inefficient.

Loki’s primary goal is to minimise the cost of switching between metrics and logs, reducing incident response time and improving user experience.

Problems with ELK

Full‑text indexing solutions like ELK provide rich features but are resource‑intensive and often overkill for simple time‑range queries, making them feel like using a sledgehammer for a nail.

Loki aims to balance query simplicity with functionality and to provide a cost‑effective alternative.

Cost Considerations

Inverted‑index based search (e.g., Elasticsearch) incurs high storage and processing costs. Alternative designs such as OKlog use grid‑based distribution to lower cost, though they may sacrifice query convenience.

Architecture

Loki uses the same label‑based indexing as Prometheus. Promtail runs as a DaemonSet on each node, discovers pods via the Kubernetes API, and forwards logs to Loki.

Components

Distributor : first receiver of logs, batches and compresses data before passing to ingesters.

Ingester : builds gzip‑compressed chunks; when a chunk is full or timed out it flushes to storage. Replication factor defaults to three.

Querier : receives time‑range and label selectors, looks up matching chunks, and streams results, supporting parallel distributed grep.

Scalability

Indexes can be stored in Cassandra, Bigtable, or DynamoDB; chunks can reside in any object store. Distributor and Querier are stateless; Ingester is stateful but rebalances chunks when nodes change.

Deployment

Installation steps (OpenShift example): create a namespace, set SCC and cluster‑admin permissions, apply a StatefulSet for Loki, a ConfigMap for Promtail, a DaemonSet for Promtail, and a NodePort Service.

oc new-project loki
oc adm policy add-scc-to-user anyuid -z default -n loki
oc adm policy add-cluster-role-to-user cluster-admin system:serviceaccount:loki:default
oc create -f statefulset.json -n loki
oc create -f configmap.json -n loki
oc create -f daemonset.json -n loki
oc create -f service.json -n loki

API and LogQL

Loki exposes an HTTP API. Example curl commands retrieve label metadata and query logs.

curl http://<host>:<port>/api/prom/label
curl http://<host>:<port>/api/prom/label/namespace/values

LogQL selectors are written inside {} with operators =, !=, =~, !~. Filter expressions such as |=, !=, |~, !~ further refine results. Regular expressions follow RE2 syntax.