Why Loki Beats ELK for Cloud‑Native Log Management

Background and Motivation

When designing a logging solution for our company's container cloud, we found that mainstream ELK or EFK stacks were heavyweight and that many of Elasticsearch's complex search features were unnecessary, so we chose Grafana's open‑source Loki system.

ELK Problems

Most log‑collection solutions index logs with full‑text search (e.g., ELK), offering rich functionality but incurring high resource usage, complexity, and operational difficulty. Many features are unused, and queries often only need a time range and simple parameters such as host or service, making ELK overkill.

Loki aims to balance query language simplicity with functionality, reducing the cost of full‑text indexing.

Cost

Full‑text search solutions like Elasticsearch have high indexing and storage costs due to inverted indexes. Alternative designs such as OKlog use eventual consistency and grid‑based distribution to lower costs, though they sacrifice query convenience. Loki’s third goal is to provide a more cost‑effective solution.

Overall Architecture

Loki’s architecture is simple and uses the same label system as Prometheus for indexing.

Labels allow querying both log content and monitoring data, reducing the cost of switching between queries and dramatically lowering storage for log indexes.

Loki uses the same service discovery and label re‑labeling as Prometheus. Promtail runs as a DaemonSet on each node, obtains metadata via the Kubernetes API, and sends logs to Loki.

Read/Write Path

Log ingestion relies on the Distributor and Ingester components.

The Distributor receives logs from Promtail. Because write volume can be large, Loki batches and compresses data using Gzip, building compressed chunks in the Ingester. When a chunk reaches a size or time threshold, it is flushed to storage, with default replication factor three.

The Ingester creates and refreshes chunks, storing them in a separate database from indexes. After flushing a chunk, a new empty chunk is created for subsequent entries.

The Querier handles reads: given a time range and label selector, it consults the index to find matching chunks and greps the results, also pulling the latest data from the Ingester.

Scalability

Loki’s index storage can use Cassandra, Bigtable, or DynamoDB, while chunks can reside in various object stores. Both Querier and Distributor are stateless. The Ingester, though stateful, redistributes chunks across nodes when the cluster changes, ensuring resilience.

Original source: http://blog.csdn.net/Linkthaha/article/details/100575651