Why Loki Beats Elasticsearch: Low Index Overhead, Fast Queries, and Easy Setup

Low Index Overhead

Loki indexes only labels, not log content, dramatically reducing indexing resource consumption compared to Elasticsearch.

Concurrent Queries & Cache

Loki splits queries into small shards, acting like parallel grep, to compensate for the lack of full‑text indexing.

Prometheus Integration

Identical label sets between Loki and Prometheus enable direct alertmanager integration.

Grafana Front‑end

Using Grafana as the UI avoids switching between Kibana and Grafana.

Architecture Overview

Loki consists of four roles:

Querier – handles HTTP queries.

Ingester – stores logs in memory.

Query‑frontend – front‑end for queries.

Distributor – distributes incoming streams.

The role can be selected via the -target binary flag.

Read Path

Querier receives HTTP requests.

It forwards queries to all ingesters to fetch in‑memory data.

If no ingester returns data, the querier loads data from the back‑end storage.

Results are deduplicated and returned over HTTP.

Write Path

The distributor receives an HTTP request to store a log stream.

Each stream is hashed and sent to the appropriate ingester replicas.

Ingester creates or appends a chunk for the stream; each tenant‑label combination has a unique chunk.

Local Installation

Download Binaries

$ wget https://github.com/grafana/loki/releases/download/v2.2.1/loki-linux-amd64.zip
$ wget https://github.com/grafana/loki/releases/download/v2.2.1/promtail-linux-amd64.zip

Configure Promtail

$ mkdir /opt/app/{promtail,loki} -p
$ cat <<EOF > /opt/app/promtail/promtail.yaml
server:
  http_listen_port: 9080
  grpc_listen_port: 0
positions:
  filename: /var/log/positions.yaml
client:
  url: http://localhost:3100/loki/api/v1/push
scrape_configs:
- job_name: system
  static_configs:
  - targets: [localhost]
    labels:
      job: varlogs
      __path__: /var/log/*.log
EOF
$ cat <<EOF > /etc/systemd/system/promtail.service
[Unit]
Description=promtail server
Wants=network-online.target
After=network-online.target
[Service]
ExecStart=/opt/app/promtail/promtail -config.file=/opt/app/promtail/promtail.yaml
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=promtail
[Install]
WantedBy=default.target
EOF
$ systemctl daemon-reload
$ systemctl restart promtail
$ systemctl status promtail

Configure Loki

$ mkdir /opt/app/{promtail,loki} -p
$ cat <<EOF > /opt/app/loki/loki.yaml
auth_enabled: false
server:
  http_listen_port: 3100
  grpc_listen_port: 9096
ingester:
  wal:
    enabled: true
    dir: /opt/app/loki/wal
  lifecycler:
    address: 127.0.0.1
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
    final_sleep: 0s
  chunk_idle_period: 1h
  max_chunk_age: 1h
  chunk_target_size: 1048576
  chunk_retain_period: 30s
  max_transfer_retries: 0
schema_config:
  configs:
  - from: 2020-10-24
    store: boltdb-shipper
    object_store: filesystem
    schema: v11
    index:
      prefix: index_
      period: 24h
storage_config:
  boltdb_shipper:
    active_index_directory: /opt/app/loki/boltdb-shipper-active
    cache_location: /opt/app/loki/boltdb-shipper-cache
    cache_ttl: 24h
    shared_store: filesystem
  filesystem:
    directory: /opt/app/loki/chunks
compactor:
  working_directory: /opt/app/loki/boltdb-shipper-compactor
  shared_store: filesystem
limits_config:
  reject_old_samples: true
  reject_old_samples_max_age: 168h
chunk_store_config:
  max_look_back_period: 0s
table_manager:
  retention_deletes_enabled: false
  retention_period: 0s
ruler:
  storage:
    type: local
    local:
      directory: /opt/app/loki/rules
  rule_path: /opt/app/loki/rules-temp
  alertmanager_url: http://localhost:9093
  ring:
    kvstore:
      store: inmemory
  enable_api: true
EOF
$ cat <<EOF > /etc/systemd/system/loki.service
[Unit]
Description=loki server
Wants=network-online.target
After=network-online.target
[Service]
ExecStart=/opt/app/loki/loki -config.file=/opt/app/loki/loki.yaml
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=loki
[Install]
WantedBy=default.target
EOF
$ systemctl daemon-reload
$ systemctl restart loki
$ systemctl status loki

Grafana Data Source

Configure Loki as a data source in Grafana and use the Explore view to query logs.

Label‑Only Indexing

Loki indexes only label sets; content is not indexed, which reduces index size dramatically.

Static Label Matching

Example promtail configuration creates a job with a fixed label job="syslog" and collects logs from /var/log/messages.

Dynamic Labels & High Cardinality

Dynamic labels have non‑fixed values; high‑cardinality labels have many possible values (e.g., IP addresses), which can create millions of streams and overwhelm Loki.

Avoid using high‑cardinality fields as labels unless necessary.

Regex Extraction in Promtail

Promtail can extract fields like action and status_code from Apache access logs using a regex stage, turning each unique combination into a separate stream.

- job_name: system
  pipeline_stages:
  - regex:
      expression: "^(?P<ip>\\S+) (?P<identd>\\S+) (?P<user>\\S+) \[(?P<timestamp>[\\w:/]+\\s[+\\-]\\d{4})\] \"(?P<action>\\S+)\\s?(?P<path>\\S+)?\\s?(?P<protocol>\\S+)?\" (?P<status_code>\\d{3}|-) (?P<size>\\d+|-)\""
  - labels:
      action:
      status_code:
  static_configs:
  - targets: [localhost]
    labels:
      job: apache
      env: dev
      __path__: /var/log/apache.log

Query Acceleration Without Labels

Use filter expressions to search non‑indexed fields, e.g., {job="apache"} |= "11.11.11.11".

Shard‑Based Query Execution

Loki splits queries into time‑range shards, opens matching chunks for each stream, and searches in parallel.

Shard size and parallelism are configurable; large deployments can process terabytes of logs in seconds.

Comparison with Elasticsearch

Elasticsearch maintains a large index regardless of query activity, consuming significant memory.

Loki performs on‑demand sharding and parallel querying, resulting in lower resource usage.

Best Practices

Keep the number of labels low for small log volumes to reduce chunk loading.

Add labels only when they improve query selectivity.

Monitor chunk target size (e.g., 1 MB) and adjust labeling strategy accordingly.

Ensure logs are ingested in chronological order; Loki rejects out‑of‑order data for performance.