Scaling ELK from 1 GB to 1 TB Daily Logs: Full‑Stack Observability Guide

1. From Log‑Explosion to Systematic Refactor

A small‑to‑medium business saw daily log volume jump from 1 GB to 1 TB, causing a cascade of ELK failures: disk usage >90%, query timeouts (>30 s), write rejections (429 Too Many Requests), and field‑count limits exceeded. The team rebuilt the stack by addressing index strategy, shard sizing, ILM, ingest processing, and operational safeguards.

2. ELK Stack Overview

ELK consists of Elasticsearch (distributed search and storage), Logstash (log collection and enrichment), and Kibana (visualization). Beats, especially Filebeat, act as lightweight collectors that can forward directly to Elasticsearch or through Logstash.

3. Core Elasticsearch Concepts

3.1 Inverted Index

Elasticsearch builds an inverted index (term → list of document IDs) enabling O(1) term look‑ups.

Term       Doc IDs
----       -------
java       [1, 2]
logging    [1]
error      [1, 3]

Query error directly returns documents 1 and 3.

3.2 Sharding and Replication

Indices are split into primary shards and optional replica shards. Shard count is fixed at index creation; it cannot be changed without reindexing. Primary shards store the original data, replicas provide high availability and read scaling.

3.3 Determining Shard Count

Rule of thumb: keep each shard 30‑50 GB. Formula: shard_count = ceil(total_index_size / 50GB) For 1 TB/day retained 15 days (≈15 TB), about 300 shards are recommended (e.g., 20 shards per daily index).

4. Index Lifecycle Management (ILM)

ILM automates hot‑warm‑cold‑delete phases:

hot : recent data on SSD, rollover at 50 GB or 1 day.

warm : less frequent writes, moved to HDD, forcemerge to 1 segment, priority 50.

cold : read‑only, stored on cheap object storage, replica count set to 0.

delete : data older than retention period is removed.

Sample ILM policy (JSON) defines rollover, transitions, and actions for each phase.

5. Index Templates and Mapping

Templates apply settings and mappings automatically when an index matches a pattern (e.g., logs-*). Key settings include number_of_shards, number_of_replicas, index.lifecycle.name, refresh_interval, and index.mapping.total_fields.limit. Mapping design: keyword for exact match, aggregations, sorting. text for full‑text search. date for timestamps.

Dynamic templates to map all strings as keyword (ignore >256 chars).

6. Log Processing Choices

6.1 Comparison

Deployment location : Filebeat Processor – collector node; Ingest Pipeline – Elasticsearch node; Logstash – separate node.

Resource usage : Filebeat Processor – very low; Ingest Pipeline – low; Logstash – high (JVM).

Complexity : Filebeat Processor – low; Ingest Pipeline – low (built‑in); Logstash – high.

Recommendation:

Simple JSON logs → Filebeat Processor.

Need GeoIP, date parsing → Ingest Pipeline.

Complex multi‑source pipelines or Kafka buffering → Logstash.

7. Kibana Features

7.1 Discover

Interactive log search with time picker, KQL/Lucene queries, field filters, and document view.

7.2 Visualize & Lens

Drag‑and‑drop visual builder (Lens) simplifies chart creation; Visualize offers a full set of chart types (bar, line, pie, metric, heat map, etc.).

7.3 Dashboard & ML

Combine visualizations into dashboards; Machine Learning jobs (single‑metric, multi‑metric, population, categorical) automatically detect anomalies such as sudden error‑rate spikes.

8. Query DSL Essentials

8.1 Match vs Term

GET logs-*/_search
{
  "query": { "match": { "message": "NullPointerException order" } }
}

match

tokenizes the query; term performs exact match on keyword fields.

8.2 Bool Queries

{
  "query": {
    "bool": {
      "must": [{"term": {"level": "ERROR"}}],
      "must_not": [{"term": {"app": "test"}}],
      "filter": [{"range": {"@timestamp": {"gte": "now-1h"}}}]
    }
  }
}

8.3 Aggregations

Bucket aggregation ( terms) and metric aggregation ( avg, percentiles) enable dashboards such as error count per app or P99 response time.

9. Performance Tuning

9.1 Write Optimizations

Increase refresh_interval (e.g., 30 s) during bulk ingest.

Temporarily set number_of_replicas to 0, then restore to 1.

Raise bulk_max_size to 1000‑5000.

Adjust indices.memory.index_buffer_size and translog.flush_threshold_size for larger buffers.

9.2 Query Optimizations

Limit time range (e.g., last hour vs last 7 days).

Use filter context to enable caching.

Avoid deep pagination; use search_after instead of from+size.

Prefer keyword fields for aggregations.

9.3 Shard‑Level Tweaks

Routing by business key (e.g., traceId) to target a single shard.

Force‑merge cold indices to a single segment.

Use shrink when shard count becomes excessive.

10. Capacity Planning

Disk requirement formula:

total_disk = raw_log_volume × (1 + replica_factor) × (1 + index_overhead) × retention_days

Example: 1 TB/day, 30 days, 1 replica, 10 % overhead → 66 TB raw, ~94 TB after applying a 70 % water‑mark.

11. High‑Availability Architecture

Deploy three dedicated master nodes (odd number) for quorum election.

Separate data roles (hot, warm, cold) to match ILM phases.

Use coordinating nodes for large queries.

Enable zone awareness attributes to spread shards across availability zones, ensuring at least one copy survives a zone failure.

12. Security Hardening

Authentication : native user/password, TLS for transport and HTTP.

RBAC : roles such as logs_reader (read‑only) and logs_writer (create docs).

TLS : mutual authentication with keystores.

Audit Logging : record access_denied, authentication_failed, etc.

Field‑level security : grant/except lists to hide sensitive fields.

Document‑level security : query‑based restrictions (e.g., owner: xiaoa).

13. End‑to‑End Deployment Walkthrough (ops‑demo)

Create Docker‑Compose stack with Elasticsearch and Kibana (single‑node for dev).

Define ILM policy ( logs-ilm-policy.json) and index template ( index-template.json).

Create the first write index and alias logs-ops-demo (write‑enabled).

Configure Filebeat to use the ingest pipeline logs-pipeline.

In Kibana, set up index pattern logs-*, explore logs, and build dashboards (error rate, top loggers, P99 latency).

14. Common Pitfalls & Mitigations

Too many shards : memory pressure; shrink or increase shard size.

Field‑count explosion : limit total_fields.limit, use dynamic templates, or flatten large JSON objects.

Deep pagination : replace from with search_after.

Replica set to 0 during ingest : remember to restore after bulk load.

Short refresh interval : increase to reduce segment churn.

Oversized JVM heap : keep ≤31 GB to stay within compressed‑oops limit.

Cluster split‑brain : run ≥3 master nodes, enable quorum election, avoid data roles on masters.

Timezone mismatch : store timestamps in UTC, configure Kibana timezone to your locale.

15. Summary & Best‑Practice Checklist

ELK = Elasticsearch + Logstash + Kibana (storage, processing, visualization).

Elasticsearch uses a Lucene inverted index (term → doc IDs) for fast full‑text search.

Plan shard size 30‑50 GB; shard count is immutable after index creation.

ILM automates hot‑warm‑cold‑delete lifecycle; typical retention 30 days.

Index templates enforce settings, mappings, and dynamic templates.

Prefer Ingest Pipeline over Logstash for simple enrichment; use Filebeat processors for ultra‑lightweight transformations.

Kibana Discover for ad‑hoc search; Visualize/Lens for charts; Dashboards for monitoring; ML for anomaly detection.

Write queries in filter context whenever possible (no _score, cacheable).

Capacity = raw × (1+replicas) × (1+overhead) × days; provision ~30 % headroom.

High availability: 3 master nodes, zone‑aware allocation, replica = 1 for hot/warm, replica = 0 for cold.

Security: RBAC, TLS, audit logs, field‑ and document‑level security.

Best‑Practice Quick‑Reference

Index strategy : daily indices + alias + ILM.

Shard sizing : 30‑50 GB per shard, compute total shards = ceil(total_size/50GB).

Field count : limit to 2000, use dynamic_templates to map strings as keyword.

Refresh interval : 30 s for log ingestion, restore to 1 s for low‑latency queries.

Replica policy : hot = 1, warm = 1, cold = 0 (save space).

Processing chain : Filebeat Processor → Ingest Pipeline → (optional) Logstash for complex pipelines.

Query design : filter context + narrow time range; avoid deep pagination; use search_after for deep scroll.

Capacity formula : raw × 2 × 1.1 × retention_days / 0.7 ≈ required disk.

Node sizing : JVM heap ≤31 GB, physical memory ≈ 2 × heap.

HA deployment : 3 master nodes, zone awareness, replica = 1 for hot/warm.

Security stack : RBAC + TLS + audit logging + field‑level security (DLS).