Deploy a Production‑Ready Loki Cluster on Kubernetes with S3 Storage

Many newcomers to Loki are confused by components such as distributor, ingester, querier, and various third‑party storage dependencies, and the official documentation provides only a brief overview of cluster deployment.

The community offers a Docker‑Compose based quick‑start for a Loki cluster, which, while not suitable for production on a single node, reveals valuable architecture and configuration insights.

The production‑grade setup differs from a pure distributed Loki cluster in three main ways:

The core services distributor, ingester, and querier run within a single instance instead of being separated.

External KV stores like Consul and etcd are abandoned; memberlist maintains cluster state in memory.

Boltdb‑shipper replaces other log indexing solutions, allowing index storage on S3 and eliminating dependencies on Cassandra or BigTable.

Beyond the mandatory S3 storage for chunks and index, an additional cache service is required to accelerate log queries and writes.

Since Loki 2.0, the boltdb‑shipper mode stores indexes in S3, simplifying horizontal scaling. See the official documentation for details.

Native Configuration

memberlist

memberlist:
  join_members: ["loki-1", "loki-2", "loki-3"]
  dead_node_reclaim_time: 30s
  gossip_to_dead_nodes_time: 15s
  left_ingesters_timeout: 30s
  bind_addr: ['0.0.0.0']
  bind_port: 7946

Loki uses the gossip protocol via memberlist to achieve eventual consistency across nodes; the defaults are generally sufficient.

ingester

ingester:
  lifecycler:
    join_after: 60s
    observe_period: 5s
    ring:
      replication_factor: 2
      kvstore:
        store: memberlist
    final_sleep: 0s

The ingester state is synchronized through gossip, and a replication factor of 2 ensures each log stream is written to two ingesters for redundancy.

Extended Configuration

storage

schema_config:
  configs:
  - from: 2021-04-25
    store: boltdb-shipper
    object_store: aws
    schema: v11
    index:
      prefix: index_
      period: 24h

storage_config:
  boltdb_shipper:
    shared_store: aws
    active_index_directory: /loki/index
    cache_location: /loki/boltdb-cache
  aws:
    s3: s3://<S3_ACCESS_KEY>:<S3_SECRET_KEY>@<S3_URL>/<S3_BUCKET>
    s3forcepathstyle: true
    insecure: true

Here, active_index_directory points to the S3 bucket path for index storage, while cache_location holds the local bolt index cache.

The ingester uploads indexes to <S3_BUCKET>/index/ .

redis

query_range:
  results_cache:
    cache:
      redis:
        endpoint: redis:6379
        expiration: 1h
  cache_results: true

index_queries_cache_config:
  redis:
    endpoint: redis:6379
    expiration: 1h

chunk_store_config:
  chunk_cache_config:
    redis:
      endpoint: redis:6379
      expiration: 1h

write_dedupe_cache_config:
  redis:
    endpoint: redis:6379
    expiration: 1h

Redis is introduced as a cache for query results and chunk storage, suitable for modest‑size clusters.

ruler

ruler:
  storage:
    type: s3
    s3:
      s3: s3://<S3_ACCESS_KEY>:<S3_SECRET_KEY>@<S3_URL>/<S3_RULES_BUCKET>
      s3forcepathstyle: true
      insecure: true
      http_config:
        insecure_skip_verify: true
    enable_api: true
    enable_alertmanager_v2: true
    alertmanager_url: "http://<alertmanager>"
    ring:
      kvstore:
        store: memberlist

The ruler service stores rule files in S3 and participates in the memberlist ring for consistent hashing.

Kubernetes Support

The complete manifest, adapted from the community’s Docker‑Compose setup, is available on GitHub and can be deployed directly to a Kubernetes cluster after configuring S3 credentials in installation.sh.

The manifest relies only on an S3 object store; ensure you have the AccessKey and SecretKey ready before installation.

The ServiceMonitor included enables Prometheus Operator metrics discovery for Loki, but its deployment is optional.

Conclusion

This article presented the official Loki production‑grade cluster deployment, enhanced with caching and S3 storage extensions, and adapted the Docker‑Compose approach for Kubernetes. The streamlined architecture reduces external dependencies and serves as a solid reference for building robust Loki clusters.