Why Choose Loki Over ELK? A Practical Guide to Scalable Log Aggregation

1. Introduction

When designing a log solution for a container cloud, the heavyweight nature of ELK (Elasticsearch, Logstash, Kibana) or EFK (Elasticsearch, Filebeat/Fluentd, Kibana) and the limited need for complex ES search features led to the selection of Grafana Loki, an open‑source log system.

The article also notes that EFK remains a mature solution worth understanding.

2. Overview

Loki is a horizontally scalable, highly available, multi‑tenant log aggregation system created by Grafana Labs. It is cost‑effective and easy to operate because it does not index log content; instead, it indexes a set of labels for each log stream, optimized for Prometheus and Kubernetes users. The project’s tagline is “Like Prometheus, but for logs.”

Project URL: https://github.com/grafana/loki/

Key characteristics of Loki compared with other log systems:

Logs are not fully indexed; only metadata is indexed, reducing cost and complexity.

Uses the same label‑based indexing as Prometheus, enabling efficient grouping and alertmanager integration.

Optimized for storing Kubernetes pod logs; pod labels are automatically indexed.

Native Grafana support eliminates the need to switch between Kibana and Grafana.

3. Architecture

3.1 Components

Promtail – log collector, analogous to Filebeat.

Loki server – analogous to Elasticsearch.

Loki processes consist of four roles, selectable via the

-target

flag:

Querier – handles query requests.

Ingester – stores incoming log data.

Query‑frontend – front‑end for queries.

Distributor – distributes writes to ingesters.

3.2 Read Path

The querier receives HTTP requests, forwards the query to all ingesters, aggregates matching data, and returns deduplicated results to the client.

3.3 Write Path

The distributor receives an HTTP request, hashes each stream, forwards it to the appropriate ingester(s) based on the configured replication factor, and the ingester creates or appends to a chunk for that stream.

4. Deployment

4.1 Local Installation

Download Promtail and Loki

<code>wget https://github.com/grafana/loki/releases/download/v2.2.1/loki-linux-amd64.zip</code>
<code>wget https://github.com/grafana/loki/releases/download/v2.2.1/promtail-linux-amd64.zip</code>

Install Promtail

<code>$ mkdir /opt/app/{promtail,loki} -pv</code>
<code># Create promtail configuration</code>
<code>$ cat <<EOF > /opt/app/promtail/promtail.yaml</code>
<code>server:</code>
<code>  http_listen_port: 9080</code>
<code>  grpc_listen_port: 0</code>
<code>positions:</code>
<code>  filename: /var/log/positions.yaml # writable by promtail</code>
<code>client:</code>
<code>  url: http://localhost:3100/loki/api/v1/push</code>
<code>scrape_configs:</code>
<code>- job_name: system</code>
<code>  static_configs:</code>
<code>  - targets:</code>
<code>    - localhost</code>
<code>    labels:</code>
<code>      job: varlogs</code>
<code>      host: yourhost</code>
<code>      __path__: /var/log/*.log</code>
<code>EOF</code>
<code># Unzip and install</code>
<code>unzip promtail-linux-amd64.zip</code>
<code>mv promtail-linux-amd64 /opt/app/promtail/promtail</code>
<code># Systemd service</code>
<code>$ cat <<EOF > /etc/systemd/system/promtail.service</code>
<code>[Unit]</code>
<code>Description=promtail server</code>
<code>Wants=network-online.target</code>
<code>After=network-online.target</code>
<code></code>
<code>[Service]</code>
<code>ExecStart=/opt/app/promtail/promtail -config.file=/opt/app/promtail/promtail.yaml</code>
<code>StandardOutput=syslog</code>
<code>StandardError=syslog</code>
<code>SyslogIdentifier=promtail</code>
<code></code>
<code>[Install]</code>
<code>WantedBy=default.target</code>
<code>EOF</code>
<code>systemctl daemon-reload</code>
<code>systemctl restart promtail</code>
<code>systemctl status promtail</code>

Install Loki

<code>$ mkdir /opt/app/{promtail,loki} -pv</code>
<code># Loki configuration</code>
<code>$ cat <<EOF > /opt/app/loki/loki.yaml</code>
<code>auth_enabled: false</code>
<code>server:</code>
<code>  http_listen_port: 3100</code>
<code>  grpc_listen_port: 9096</code>
<code>ingester:</code>
<code>  wal:</code>
<code>    enabled: true</code>
<code>    dir: /opt/app/loki/wal</code>
<code>  lifecycler:</code>
<code>    address: 127.0.0.1</code>
<code>    ring:</code>
<code>      kvstore:</code>
<code>        store: inmemory</code>
<code>      replication_factor: 1</code>
<code>    final_sleep: 0s</code>
<code>  chunk_idle_period: 1h</code>
<code>  max_chunk_age: 1h</code>
<code>  chunk_target_size: 1048576</code>
<code>  chunk_retain_period: 30s</code>
<code>  max_transfer_retries: 0</code>
<code>schema_config:</code>
<code>  configs:</code>
<code>    - from: 2020-10-24</code>
<code>      store: boltdb-shipper</code>
<code>      object_store: filesystem</code>
<code>      schema: v11</code>
<code>      index:</code>
<code>        prefix: index_</code>
<code>        period: 24h</code>
<code>storage_config:</code>
<code>  boltdb_shipper:</code>
<code>    active_index_directory: /opt/app/loki/boltdb-shipper-active</code>
<code>    cache_location: /opt/app/loki/boltdb-shipper-cache</code>
<code>    cache_ttl: 24h</code>
<code>    shared_store: filesystem</code>
<code>  filesystem:</code>
<code>    directory: /opt/app/loki/chunks</code>
<code>compactor:</code>
<code>  working_directory: /opt/app/loki/boltdb-shipper-compactor</code>
<code>  shared_store: filesystem</code>
<code>limits_config:</code>
<code>  reject_old_samples: true</code>
<code>  reject_old_samples_max_age: 168h</code>
<code>chunk_store_config:</code>
<code>  max_look_back_period: 0s</code>
<code>table_manager:</code>
<code>  retention_deletes_enabled: false</code>
<code>  retention_period: 0s</code>
<code>ruler:</code>
<code>  storage:</code>
<code>    type: local</code>
<code>    local:</code>
<code>      directory: /opt/app/loki/rules</code>
<code>  rule_path: /opt/app/loki/rules-temp</code>
<code>  alertmanager_url: http://localhost:9093</code>
<code>  ring:</code>
<code>    kvstore:</code>
<code>      store: inmemory</code>
<code>  enable_api: true</code>
<code>EOF</code>
<code># Unzip and install</code>
<code>unzip loki-linux-amd64.zip</code>
<code>mv loki-linux-amd64 /opt/app/loki/loki</code>
<code># Systemd service</code>
<code>$ cat <<EOF > /etc/systemd/system/loki.service</code>
<code>[Unit]</code>
<code>Description=loki server</code>
<code>Wants=network-online.target</code>
<code>After=network-online.target</code>
<code></code>
<code>[Service]</code>
<code>ExecStart=/opt/app/loki/loki -config.file=/opt/app/loki/loki.yaml</code>
<code>StandardOutput=syslog</code>
<code>StandardError=syslog</code>
<code>SyslogIdentifier=loki</code>
<code></code>
<code>[Install]</code>
<code>WantedBy=default.target</code>
<code>EOF</code>
<code>systemctl daemon-reload</code>
<code>systemctl restart loki</code>
<code>systemctl status loki</code>

5. Usage

5.1 Configure Loki datasource in Grafana

In Grafana, add a new datasource of type Loki and set the URL to

http://loki:3100

. After saving, open the Explore view to query logs.

5.2 Query logs in Grafana Explore

<code>rate({job="message"} |= "kubelet" [1m])</code>

5.3 Indexing only labels

Loki indexes only the label set, not the full log content. Example Promtail configuration shows a job with a fixed label

job="syslog"

and path

/var/log/messages

. Multiple jobs can be queried with regex label matchers such as

{job=~"apache|syslog"}

.

5.4 Label matching characteristics

Labels work like Prometheus series: a unique combination of label values creates a stream; when any label changes, a new stream (hash) is generated.

5.5 Dynamic labels and high cardinality

Dynamic labels have non‑fixed values; high‑cardinality labels have many possible values (e.g., IP addresses). Using high‑cardinality labels can create a massive number of streams, which may overwhelm Loki.

5.6 Full‑text indexing considerations

Full‑text indexes are large and costly. Loki’s index is typically an order of magnitude smaller than the ingested log volume, growing slowly.

5.7 Query sharding

Loki splits queries into time‑based shards, opens the relevant chunks for matching streams, and processes them in parallel, allowing high‑throughput log queries.

5.8 Best practices

Keep the number of labels low for small log volumes to reduce chunk loading.

Add labels only when needed; high‑cardinality labels should be avoided.

Ensure logs are ingested in time‑ascending order; Loki rejects old data for performance.