Why Loki Beats ELK for Kubernetes Logging: Architecture, Deployment, and Query Guide

Background

When designing a log solution for a container cloud, the team found traditional ELK/EFK stacks too heavy and feature‑rich for their needs, so they chose the open‑source Grafana Loki system.

Motivation

In a Kubernetes environment, Pods write logs to stdout and stderr. When a Pod’s memory spikes and triggers an alert, administrators must inspect the Pod’s logs to diagnose the issue. Without a log aggregation system, this requires manual access via the UI or CLI, which is cumbersome.

Loki’s primary goal is to minimise the cost of switching between metrics and logs, reducing incident response time and improving user experience.

Problems with ELK

Full‑text indexing solutions like ELK provide rich functionality but are resource‑intensive, complex to operate, and often include features that are unnecessary for simple time‑range and label‑based queries, making them a case of "using a cannon to kill a mosquito".

Cost Considerations

Full‑text search incurs high storage and processing costs due to inverted index creation. Alternative designs such as OKlog use grid‑based distribution to lower costs but sacrifice query convenience. Loki aims to balance cost‑effectiveness with ease of use.

Overall Architecture

Loki’s architecture mirrors Prometheus: it uses the same label‑based indexing, allowing users to query logs and metrics with the same selectors, dramatically reducing storage overhead.

Promtail runs as a DaemonSet on each node, discovers Pods via the Kubernetes API, attaches metadata, and forwards logs to Loki.

Read/Write Path

Logs are written through the Distributor and Ingester components. The Distributor receives logs from Promtail, hashes them, and forwards them to an appropriate Ingester. Ingester compresses logs into gzip chunks and flushes them to storage when a chunk reaches a size or time threshold. Chunks are replicated (default three times) for redundancy.

After flushing, Ingester creates a new empty chunk for subsequent entries.

Querier

The Querier receives a time range and label selector, looks up matching chunks via the index, and streams results back, optionally merging in‑flight data from Ingester. Queries are parallelised across distributed chunks.

Scalability

Loki can store indexes in Cassandra, Bigtable, or DynamoDB, while chunks reside in object storage. Distributor and Querier are stateless; Ingester is stateful but rebalances chunks when nodes are added or removed. The underlying storage implementation (Cortex) has been proven in production.

Deployment

Installation on OpenShift/Kubernetes is straightforward. oc new-project loki Set permissions:

oc adm policy add-scc-to-user anyuid -z default -n loki
oc adm policy add-cluster-role-to-user cluster-admin system:serviceaccount:loki:default

Deploy Loki: oc create -f statefulset.json -n loki Deploy Promtail: oc create -f configmap.json -n loki Deploy Promtail DaemonSet: oc create -f daemonset.json -n loki Create Service:

oc create -f service.json -n loki

API Overview

Loki exposes an HTTP API (see GitHub for details). The following examples illustrate how to query metadata and logs.

Step 1 – Get label names

curl http://<em>HOST</em>:<em>PORT</em>/api/prom/label

Step 2 – Get values for a label

curl http://<em>HOST</em>:<em>PORT</em>/api/prom/label/namespace/values

Step 3 – Query logs

http://<em>HOST</em>:<em>PORT</em>/api/prom/query?direction=BACKWARD&limit=1000&query={namespace="cicd"}&start=1567644457221000000&end=1567730857221000000&refId=A

Query Parameters

query : LogQL selector and filter, e.g. {name=~"mysql.+"} |= "error" limit : Maximum number of log lines to return

start : Start timestamp (nanoseconds since epoch)

end : End timestamp (nanoseconds since epoch)

direction : FORWARD or BACKWARD (default BACKWARD)

regexp : Optional regex to filter results

LogQL Syntax

Selectors

Label matchers are placed inside {} and separated by commas. Supported operators: = – exact match != – not equal =~ – regex match !~ – regex not match

Filter Expressions

After a selector, use filter operators to further narrow results: |= – line contains string != – line does not contain string |~ – line matches regex !~ – line does not match regex

Multiple filters can be combined, e.g. {job="mysql"} |= "error" != "timeout". Regular expressions follow the RE2 syntax.

Author: 小虫