Why Loki Beats ELK for Cloud‑Native Log Management

Background and Motivation

When designing a logging solution for a container cloud, the heavyweight nature of mainstream ELK or EFK stacks and the limited need for complex Elasticsearch search features led to the selection of Grafana's open‑source Loki system.

Problems with ELK

Full‑text indexing solutions like ELK provide rich functionality but are resource‑intensive and often include features that go unused; most queries only need a time range and simple parameters such as host or service, making ELK feel like overkill.

Cost

Full‑text search incurs high costs due to inverted index creation and sharing; alternative designs like OKlog reduce cost and simplify operations but sacrifice query convenience, prompting Loki to aim for a more cost‑effective solution.

Overall Architecture

Loki uses the same label‑based indexing as Prometheus, allowing logs and metrics to be queried together, reducing switch costs and storage overhead. Promtail runs as a DaemonSet on each node, gathers logs via the Kubernetes API, and forwards them to Loki.

Read/Write

Log writes rely on the Distributor and Ingester components. The Distributor receives logs from Promtail and batches/compresses them, while the Ingester builds compressed chunks, replicates them for redundancy, and flushes them to storage.

Distributor

The Distributor is the first component to receive logs; it batches and compresses data to avoid overwhelming the database.

Ingester

The Ingester constructs chunks by compressing incoming logs; once a chunk reaches a size or time threshold, it is flushed to the database, after which a new empty chunk is created for further entries.

Querier

Queries are handled by the Querier, which uses a time range and label selector to locate matching chunks, performs distributed greps, and also retrieves the latest unflushed data from Ingesters.

Scalability

Loki’s index storage can be backed by Cassandra, Bigtable, or DynamoDB, while chunks reside in object storage; both Querier and Distributor are stateless, and Ingester state is rebalanced when nodes are added or removed, leveraging the Cortex storage engine proven in production.