Cutting Log Storage Costs 70% for 100 Billion Daily Logs: A Full Guide to Hot‑Cold Separation Architecture

Why Large‑Scale Log Systems Need Hot‑Cold Separation

Most teams start with a simple pipeline: JSON logs → Filebeat/Fluent Bit → Kafka → Elasticsearch, retaining 7–15 days. When QPS reaches 120 k and daily logs hit 90–120 billion entries, three walls appear: write amplification, query‑write resource contention, and exploding storage cost because hot and old data share the same high‑performance media.

The core contradiction is that we must simultaneously satisfy:

sub‑second queries for the last 3 days,

acceptable latency for up to 30 days of troubleshooting,

audit‑grade retention for 180 days,

no impact on production services, and

cost that does not grow linearly with data volume.

These constraints drive the hot‑cold separation design.

Problem Essence: Why Logs Misbehave Under High Concurrency

2.1 Logs Are High‑Throughput Time‑Series Data

Characteristics include append‑only writes, strong timestamps, semi‑stable fields, clear time locality, and mixed query patterns (exact filters, fuzzy search, aggregations). This makes logs a hybrid system spanning streaming ingestion, online search, offline archiving, and historical analysis.

2.2 Cost Is Driven by Indexes and Replicas, Not Raw Size

Storing raw 8 TB/day in Elasticsearch inflates to >2.5× because of JSON redundancy, inverted‑index bloat, doc values, segment merges, and replica storage on SSDs.

2.3 Query Slowness Often Comes from Bad Query Paths

80 % of queries target the recent 24 h–7 days and filter on fields like service, traceId, level, host, keyword. If all queries hit the hot cluster, they compete for CPU, I/O, and page cache, causing low‑value long‑range scans to throttle high‑value real‑time queries.

2.4 The Theory Behind Hot‑Cold Separation

Recent data is accessed frequently and is high‑value; older data is accessed rarely but may still need to be retained. Storage cost should be based on access value, not retention time.

Thus a three‑layer goal emerges:

Hot layer: guarantee write success, low latency, and stability for recent data.

Warm layer (optional): absorb mid‑term data.

Cold layer: store long‑term data at minimal cost while remaining queryable.

Target Architecture Overview

The recommended stack is:

Collection: Fluent Bit / Filebeat on each pod.

Buffer: Kafka for peak‑shaving and replay.

Hot storage: Elasticsearch (or ClickHouse) for the last 3–7 days.

Cold storage: Object storage (S3/MinIO) with Parquet files.

Query layer: Trino / Presto for cold data, unified behind a query gateway.

Governance: ILM, audit, rate‑limiting, and cost dashboards.

4.1 Component Responsibilities

Collection layer : lightweight, back‑pressure aware, DaemonSet‑deployed, enriches logs with Kubernetes metadata.

Buffer layer : Kafka topics with enough partitions to match write throughput (e.g., 96 partitions for 1.8 M logs/s, assuming 20‑30 k per partition).

Hot layer : stores recent logs, uses data streams or daily indices, limits shard size to 30‑50 GB, disables unnecessary field indexing, and separates hot nodes from coordinating nodes.

Archive layer : periodically exports hot indices to Parquet, records metadata, and validates row counts and checksums.

Cold layer : Parquet files partitioned by date and service, queried via Trino with predicate push‑down.

Unified query gateway : authenticates, validates parameters, routes time ranges to hot or cold engines, merges results, enforces rate limits, and logs audit trails.

4.3 Why Object Storage + Parquet

Parquet offers >70 % compression with Snappy/ZSTD, column‑only reads, effective predicate push‑down, and seamless integration with Trino, Spark, Flink, Iceberg, etc. It turns the “seconds‑level problem” solved by Elasticsearch into a “low‑cost, searchable archive”.

Key Design 1 – Hot Layer Stability

5.1 Collection Strategy

Fluent Bit config (excerpt):

# fluent-bit.conf
[SERVICE]
    Flush        1
    Grace        30
    Log_Level    info
    storage.path /var/fluent-bit/state
    storage.backlog.mem_limit 512MB
[INPUT]
    Name   tail
    Path   /var/log/containers/*.log
    Parser cri
    Tag    kube.*
    Mem_Buf_Limit 100MB
    Skip_Long_Lines On
[OUTPUT]
    Name   kafka
    Match  kube.*
    Brokers kafka-0:9092,kafka-1:9092,kafka-2:9092
    Topics app-logs
    Timestamp_Key timestamp
    Timestamp_Format iso8601
    rdkafka.compression.codec lz4
    rdkafka.request.required.acks 1
    rdkafka.queue.buffering.max.ms 50

Rule of thumb : the collector should only transport logs, not perform heavy parsing or routing.

5.2 Kafka Partition Planning

Target write rate: 1.8 M logs/s. Assuming 20‑30 k per partition, allocate ~96 partitions (rounded up for redundancy). Key the partition by service+pod or service+traceId to avoid hotspot skew.

5.3 Hot Storage Choice: Elasticsearch vs ClickHouse

Elasticsearch excels at full‑text search and developer debugging; ClickHouse offers higher compression and faster aggregations. The article keeps Elasticsearch as the primary hot store but provides a ClickHouse migration path.

5.4 Production‑Grade Elasticsearch Settings

Key settings (simplified):

PUT _index_template/logs-hot-template
{
  "index_patterns": ["logs-hot-*"],
  "template": {
    "settings": {
      "number_of_shards": 12,
      "number_of_replicas": 1,
      "refresh_interval": "10s",
      "codec": "best_compression",
      "routing.allocation.require.data_tier": "hot",
      "mapping.total_fields.limit": 300,
      "index.lifecycle.name": "logs-hot-ilm"
    },
    "mappings": {
      "dynamic": false,
      "properties": {
        "@timestamp": {"type": "date"},
        "service": {"type": "keyword"},
        "namespace": {"type": "keyword"},
        "host": {"type": "keyword"},
        "pod": {"type": "keyword"},
        "traceId": {"type": "keyword"},
        "level": {"type": "keyword"},
        "message": {
          "type": "text",
          "analyzer": "standard",
          "fields": {"raw": {"type": "keyword", "ignore_above": 256}}
        },
        "stack": {"type": "text", "index": false},
        "ext": {"type": "flattened"}
      }
    }
  }
}

Important points:

Disable dynamic mapping to prevent field explosion.

Store rarely queried large fields (e.g., stack traces) as text with index: false.

Keep shard size 30‑50 GB and refresh interval around 10 s for bulk ingestion.

Key Design 2 – Reliable Archiving

6.1 Archive State Machine

States:

INIT → FREEZE_WRITE → EXPORTING → EXPORTED → VALIDATING → ARCHIVED → DELETING_HOT → DONE

. Each state is persisted in a metadata table so the process can be retried, audited, and rolled back.

6.2 Consistency Strategy

Only delete hot indices after the export is validated. The workflow:

Detect index exceeds hot retention.

Ensure Kafka lag is zero.

Freeze the index (read‑only).

Export to object storage.

Write archive metadata.

Validate counts and checksums.

Delete hot data.

Mark DONE or retry on failure.

6.3 Parquet Partitioning

Partition by date ( dt=YYYY‑MM‑DD) and service, optionally by namespace. Example layout:

s3://log-archive/
  dt=2026-05-01/
    service=trade-gateway/part-0001.snappy.parquet
    service=order-service/part-0001.snappy.parquet

Avoid over‑partitioning; keep file size around 800 k records (configurable via Spark maxRecordsPerFile).

6.4 Spark Archiving Job (Python)

import datetime, json
from pyspark.sql import SparkSession
from pyspark.sql.functions import col, date_format, lit

HOT_RETENTION_DAYS = 7
ARCHIVE_BUCKET = "s3a://log-archive"
TARGET_DATE = datetime.date.today() - datetime.timedelta(days=HOT_RETENTION_DAYS)
INDEX_NAME = f"logs-hot-{TARGET_DATE.strftime('%Y.%m.%d')}"
ARCHIVE_PATH = f"{ARCHIVE_BUCKET}/dt={TARGET_DATE.strftime('%Y-%m-%d')}"

spark = (SparkSession.builder
    .appName("log-archive-job")
    .config("spark.sql.adaptive.enabled", "true")
    .config("spark.sql.shuffle.partitions", "256")
    .config("spark.sql.files.maxRecordsPerFile", "800000")
    .getOrCreate())

# mark index read‑only, get source count … (omitted for brevity)

df = spark.read.format("org.elasticsearch.spark.sql")\
    .option("es.nodes", "es-hot-client")\
    .option("es.port", "9200")\
    .option("es.resource", INDEX_NAME)\
    .load()

archive_df = (df.select(
    col("@timestamp").alias("timestamp"),
    col("service"), col("namespace"), col("level"),
    col("traceId"), col("host"), col("message"), col("costMs"))
    .withColumn("dt", date_format(col("timestamp"), "yyyy-MM-dd")))

(archive_df.repartition(128, "service")
    .sortWithinPartitions("service", "timestamp")
    .write.mode("overwrite")
    .partitionBy("dt", "service")
    .format("parquet")
    .option("compression", "snappy")
    .save(ARCHIVE_BUCKET))

The job validates that the archived row count matches the source count before writing metadata.

Key Design 3 – Unified Query Gateway

The gateway hides hot/cold differences. It performs:

Authentication & tenant isolation.

Parameter validation.

Time‑range routing (hot, cold, or mixed).

Result merging and global sorting.

Rate‑limiting, circuit‑breaking, and async handling for long‑running cold queries.

Audit logging.

Routing Logic Example (Java)

@Service
public class LogQueryFacade {
    private static final Duration HOT_WINDOW = Duration.ofDays(7);
    private final EsLogQueryService esService;
    private final TrinoLogQueryService trinoService;
    private final QueryMergeService mergeService;
    private final QueryGuardService guardService;

    public QueryResponse search(String user, QueryRequest request) {
        guardService.validate(user, request);
        Instant hotBoundary = Instant.now().minus(HOT_WINDOW);
        Instant start = Instant.ofEpochMilli(request.startTime());
        Instant end = Instant.ofEpochMilli(request.endTime());
        if (!start.isBefore(hotBoundary)) {
            return QueryResponse.sync(esService.search(request));
        }
        if (end.isBefore(hotBoundary)) {
            return handleColdQuery(request);
        }
        QueryRequest hotReq = request.withTimeRange(hotBoundary.toEpochMilli(), request.endTime());
        QueryRequest coldReq = request.withTimeRange(request.startTime(), hotBoundary.toEpochMilli() - 1);
        SearchResult hot = esService.search(hotReq);
        SearchResult cold = trinoService.search(coldReq);
        SearchResult merged = mergeService.mergeByTimestampDesc(hot, cold, request.page(), request.size());
        return QueryResponse.sync(merged);
    }

    private QueryResponse handleColdQuery(QueryRequest request) {
        if (guardService.shouldAsync(request)) {
            String queryId = trinoService.submitAsync(request);
            return QueryResponse.async(queryId);
        }
        return QueryResponse.sync(trinoService.search(request));
    }
}

Result Merge Service (Java)

@Service
public class QueryMergeService {
    public SearchResult mergeByTimestampDesc(SearchResult hot, SearchResult cold, int page, int size) {
        List<LogView> merged = new ArrayList<>(hot.items().size() + cold.items().size());
        int i = 0, j = 0;
        while (i < hot.items().size() && j < cold.items().size()) {
            LogView left = hot.items().get(i);
            LogView right = cold.items().get(j);
            if (left.timestamp() >= right.timestamp()) {
                merged.add(left);
                i++;
            } else {
                merged.add(right);
                j++;
            }
            if (merged.size() >= page * size) break;
        }
        while (i < hot.items().size() && merged.size() < page * size) merged.add(hot.items().get(i++));
        while (j < cold.items().size() && merged.size() < page * size) merged.add(cold.items().get(j++));
        int from = Math.max((page - 1) * size, 0);
        int to = Math.min(from + size, merged.size());
        List<LogView> pageItems = from >= to ? List.of() : merged.subList(from, to);
        return new SearchResult(pageItems, hot.total() + cold.total());
    }
}

Async Cold Query Service (Java)

@Service
public class AsyncColdQueryService {
    private final ExecutorService executor = Executors.newFixedThreadPool(16);
    private final ConcurrentMap<String, QueryTaskResult> taskStore = new ConcurrentHashMap<>();
    private final TrinoLogQueryService trinoService;

    public String submit(QueryRequest request) {
        String taskId = UUID.randomUUID().toString();
        taskStore.put(taskId, QueryTaskResult.pending());
        executor.submit(() -> {
            try {
                SearchResult result = trinoService.search(request);
                taskStore.put(taskId, QueryTaskResult.success(result));
            } catch (Exception ex) {
                taskStore.put(taskId, QueryTaskResult.failed(ex.getMessage()));
            }
        });
        return taskId;
    }

    public QueryTaskResult get(String taskId) {
        return taskStore.getOrDefault(taskId, QueryTaskResult.notFound());
    }
}

Production implementations would persist task state to Redis or a DB and add TTL.

Governance, Observability, and Cost Control

Key governance items include:

Fine‑grained RBAC and row‑level filtering by tenant, namespace, and service.

Field‑level masking for sensitive data (phone, ID, token).

Rate limits per user/group, maximum time span, and max result size.

Audit logs for every query.

Metrics such as log_ingest_qps, kafka_topic_lag, es_bulk_latency_p95, archive_job_duration_seconds, cold_query_latency_p95, etc.

Two dashboards: stability (write success, Kafka lag, ES bulk latency, archive success) and cost (hot SSD usage, object storage volume, top services by log volume, low‑value field bloat).

Cost Modeling – How the 70 % Savings Are Calculated

Assumptions:

Raw logs: 8 TB/day → 1440 TB for 180 days.

ES expansion factor: 2.5×.

Parquet compression: 25 % of raw JSON.

ES replica count: 1 (2× storage).

Pure‑hot solution would need 7200 TB of SSD (1440 TB × 2.5 × 2).

Hot‑cold separation reduces hot storage to ~280 TB (7 days × 8 TB × 2.5 × 2) and cold storage to ~346 TB of cheap object storage (173 days × 8 TB × 0.25). The overall yearly cost drops to roughly 29 % of the pure‑hot baseline – a 71 % reduction.

Real‑world numbers from a production rollout:

Hot nodes: 14 vs 30 in the pure‑hot design.

SSD capacity: 190 TB vs 420 TB.

Object storage: 380 TB added, but at a fraction of SSD cost.

Annual cost: ~0.29× the original, i.e., 71 % savings.

Common Pitfalls and Recommendations

Do not rely solely on Elasticsearch ILM for long‑term retention; it still keeps data on expensive nodes.

Never run archiving scripts without a state machine, metadata table, and alerting – they become black boxes.

Avoid synchronous execution of all cold queries; use async APIs, pagination limits, and result export.

Provide a reliable back‑fill (re‑ingest) path for cold‑to‑hot recovery (audit, legal, or bug‑fix scenarios).

Govern log sources: enforce schema, truncate oversized fields, whitelist dynamic fields, and disable debug logging in production.

Roadmap – From First Implementation to a Unified Observability Platform

Stage 1 : Stabilize Kafka buffering, hot ES/ClickHouse, object‑store cold layer, and unified query gateway.

Stage 2 : Add rate‑limiting, async cold queries, RBAC, field masking, small‑file handling, and cost dashboards.

Stage 3 : Introduce a warm layer, replace hot ES with ClickHouse, adopt Iceberg for cold tables, and build back‑fill tooling.

Stage 4 : Merge logs, metrics, and traces into a single observability data platform with trace‑based cross‑system navigation.

Conclusion

Hot‑cold separation is not a storage trick but a layered architecture that aligns storage cost with access value, guarantees write stability, provides searchable history, and offers a unified developer experience. By combining Kafka buffering, Elasticsearch/ClickHouse for hot data, Parquet on object storage for cold data, and a query gateway that merges results, teams can handle billions of daily logs while cutting storage costs by over 70 % and improving operational reliability.