Designing a Production‑Grade Distributed Logging and Metrics Platform

Design a Distributed Logging and Metrics Platform

System Requirements

Functional Requirements

Near‑real‑time ingestion from multiple sources (services, SDKs, agents, IoT devices) of logs, metrics, and events.

Support for structured, semi‑structured, and unstructured text data.

Parsing, transformation, enrichment of logs/events with metadata such as user_id, environment, version.

Full‑text log search with filters on timestamp, service, error level, etc.

Metric queries across time ranges and aggregation windows.

SQL‑like queries via Presto, Trino, or Spark SQL over event datasets.

Tag‑based routing (e.g., org_id, region, event_type) to different processing paths or aggregation points.

Threshold alerts and anomaly detection (e.g., via Grafana or Prometheus).

Public APIs and dashboards for log and metric consumption.

Cold‑storage archiving for compliance or audit purposes.

Non‑Functional Requirements

Handle millions of events per second with end‑to‑end latency < 100 ms.

Cross‑data‑center and cross‑region fault tolerance with no single point of failure.

Horizontal scalability at every layer (ingest, stream processing, storage, query).

TLS and KMS encryption for data at rest and in transit.

Fine‑grained RBAC with audit logging of accesses.

Self‑healing, auto‑scaling, and observability of the platform itself.

Separate hot storage (fast access) and cold storage (cost‑effective).

Support for template upgrades, data validation, and template registry integration.

Cloud‑agnostic or hybrid‑cloud compatibility for global workloads.

Capacity Estimation

Input Rate

Peak input: 2 M events/second.

Daily volume: ~150 B events/day.

Log Size

Average log/event size: 1.2 KB.

Daily total input: ~180 TB/day.

Storage Needs

Hot storage (7‑day retention): ~1.2 PB.

Cold storage (90 days, assuming Parquet compression 2:1): ~16 PB.

Kafka Capacity and Alignment

Kafka serves as the backbone separating ingestion from downstream processing.

Why Kafka? High throughput, durability, built‑in replication, ecosystem integration, at‑least‑once delivery, horizontal partitioning, and fine‑grained replay.

Configuration Details:

Topic partitions: ~15 000.

To safely handle 2 M events/s (≈2 400 MB/s) keep partition load < 10 MB/s → minimum 240 partitions, scalable to 15 000 for multi‑tenant, retry, and parallel processing.

Broker count: ~150.

Each partition processes ~100 active partitions; log segments stored on NVMe SSD.

Replication factor: 3 (cross‑rack HA).

Retention: hot logs kept 3 days, then downstream ETL or cold archiving.

Back‑pressure: Kafka provides bounded queues and flow control via consumer lag monitoring.

Stream Processing (Apache Flink)

Flink is used for real‑time, high‑throughput stream processing such as enrichment, transformation, windowed aggregation, anomaly detection, and routing.

Why Flink?

Native event‑time handling (watermarks) for out‑of‑order data.

Low‑latency streaming with high throughput and fine‑grained checkpoints.

RocksDB state backend provides fault tolerance and exactly‑once guarantees.

Rich CEP, async I/O, and broadcast state support.

Configuration:

Operators parallelism 20–50 based on CPU/memory usage.

Event‑time support with watermarks for out‑of‑order windows.

Checkpointing: RocksDB state backend + S3 checkpoints (≈100 TB total state).

Recovery SLA: job manager < 2 min, otherwise job fails.

Use cases: multi‑tenant routing, anomaly detection, enrichment.

Elasticsearch Cluster

Elasticsearch powers full‑text log search and structured field queries.

Why Elasticsearch?

Mature, production‑validated full‑text engine.

Real‑time indexing and querying of nested JSON documents.

Kibana integration for visual dashboards.

Configuration:

Raw log ingest: ~100 TB/day (uncompressed).

Index mapping includes text + JSON fields.

Cluster layout: 300 hot nodes, 100 warm nodes.

Retention policies:

Hot data (7 days) on fast SSD.

Warm data (30 days) on slower EBS.

Cold data snapshots to S3 or GCS.

Sharding by tenant or service with daily rollover indices.

Object Storage (S3 / GCS) Data Lake

Batch analytics, long‑term retention, and ad‑hoc exploration are handled via a data lake.

Why Object Storage + Lakehouse?

Cheap, durable, virtually unlimited scaling.

Compatible with major engines (Spark, Trino, Presto, Hive).

Decoupling compute from storage reduces cost.

Configuration:

Daily input: ~30 TB/day (Snappy‑compressed Parquet).

Partitioning by org_id, event_type, dt.

Query engines: Presto, Trino, Athena, BigQuery.

Lifecycle policy: 90 days hot, then cold/archive, delete after 1 year for compliance.

Time‑Series Database (Metrics)

Prometheus‑compatible back‑ends (e.g., Mimir, Cortex, VictoriaMetrics) provide low‑latency metric ingestion and alerting.

Why a TSDB?

Built‑in aggregation, compression, down‑sampling.

Designed for high cardinality and dimensional labeling.

Tight integration with Grafana and PromQL.

Configuration:

Sample rate: 8 M samples/second.

Cardinality: 5 M unique series (via labels).

Retention: 30 days raw → down‑sampled tiers (1 m, 5 m, 1 h).

Storage estimate: ~100 TB/month after compression.

Use cases: SLO dashboards, Grafana panels, anomaly detection.

Design Trade‑offs

Kafka vs. Kinesis / Pulsar

Kafka chosen for maturity, larger community, superior replay control.

Pulsar offers native tiered storage, but Kafka + S3 connector achieves similar effect.

Elasticsearch vs. OpenSearch vs. Loki

Elasticsearch selected for higher maturity and ecosystem support (Kibana).

Loki is cheaper for pure log workloads but lacks full‑text search.

OpenSearch is an open‑source alternative but may lack commercial support and stability in some versions.

Flink vs. Spark Streaming vs. Kafka Streams

Flink provides native event handling, better stateful scaling, and higher throughput.

Spark is optimized for batch; Kafka Streams is too simple for multi‑stage pipelines.

Data Lake vs. Traditional Databases

Data lakes offer scale‑economics, schema‑on‑read, and can handle massive, diverse formats.

Lakehouse formats (Iceberg, Hudi, Delta) add indexing and stability to native lakes.

TSDB vs. InfluxDB / OLAP

Prometheus‑compatible back‑ends fit Kubernetes‑native workloads.

OLAP engines like Druid or ClickHouse complement long‑term aggregation.

mTLS, RBAC & KMS

mTLS ensures mutual authentication between internal services.

RBAC provides role‑based API and UI access policies.

KMS manages encryption keys for S3, Elasticsearch, and databases.

Data Flow Explanation

1. Client / Application Layer

Services, SDKs, and agents send logs/metrics/events to ingestion endpoints.

2. Kafka Layer

Messages land on partitioned Kafka topics (keyed by org_id, env, etc.).

Kafka buffers traffic, supports retries, and enables fan‑out.

3. Flink Processing

Flink reads Kafka data, enriches, filters, and routes to Elasticsearch, S3, or the metrics DB.

Transformations include flattening, JSON schema repair, and anomaly detection.

4. Elasticsearch

Rich logs are indexed with structured and unstructured fields.

Kibana exposes APIs and dashboards.

5. Data Lake

Flink/Spark batch jobs push raw or aggregated logs to Parquet files on S3.

Trino/Presto/Athena perform partition‑pruned queries.

6. Time‑Series Database

Prometheus‑compatible scrapers or exporters push system metrics.

Real‑time dashboards and alerts are supported.

7. Cold Storage / Archival

After retention windows close, data is archived to Glacier or Deep Archive layers.

8. Access Layer

APIs, dashboards, and SQL engines serve data to users and downstream systems.

Deep Q&A

Question 1. How to scale Kafka beyond 15,000 partitions?

Use multiple Kafka clusters sharded by organization/environment.

Tier old messages to cold storage.

Adopt KRaft mode for a more scalable controller.

Question 2. What happens if a Flink checkpoint fails? How to recover?

Flink rolls back to the last successful checkpoint.

Fix the root cause (e.g., disk overload, corrupted state) and restart the job with HA enabled.

Question 3. How to ensure idempotency between Kafka and downstream consumers?

Deduplicate using Kafka message keys.

Apply a consistent unique event_id and track it in downstream sinks (ES or DB).

Question 4. Why choose Elasticsearch over OpenSearch?

More stable ecosystem and better observability integrations.

Commercial support available; OpenSearch offers a more permissive license but less mature support.

Question 5. How to partition Kafka topics for multi‑tenant isolation?

Topic naming pattern: log.org_<org_id>.env_<env> Each tenant gets its own topic with ACLs enforcing access control.

Question 6. What is the back‑pressure handling between Kafka → Flink → ES?

Tune Flink buffer timeouts and async checkpoints.

Monitor Kafka consumer lag and Flink operator metrics.

Adjust ES bulk size and retry policies.

Question 7. How to implement GDPR‑compliant deletion in ES and S3?

Use document‑level TTL or tags in ES lifecycle policies.

For S3, partition by organization/date and use Lambda to delete based on request logs.

Question 8. If log peak spikes to 5 M/s, what is the first bottleneck and how to fix it?

Kafka disk/network I/O on SSD‑based brokers – add partitions.

Flink back‑pressure – scale job manager/task managers.

ES bulk indexing queue saturation – scale ES data nodes, tune JVM heap and queue settings.

Ops Q1. How to enable mTLS between services? Which tool to rotate certificates?

Use Istio or Consul Connect for automatic mTLS.

Certificate manager with Vault or ACM for issuance and rotation.

Ops Q2. What are the SLOs for log ingestion and search latency?

Ingestion: p95 log latency < 2 s, metric latency < 1 s.

Search: recent logs p95 < 1 s, 30‑day retained logs ~64 s.

Ops Q3. How to auto‑scale Flink jobs based on traffic?

Monitor Kafka lag and Flink operator utilization.

Use Flink reactive scaling mode together with Kubernetes HPA.

Ops Q4. How to benchmark and choose NVMe vs. EBS for the ES cluster?

Run load tests for various query/write workloads.

NVMe offers higher IOPS for high‑QPS clusters; switch to EBS for cost‑sensitive workloads.

Advanced Q1. How to join Kafka logs with metadata stored in a database (e.g., user info)?

Use Flink asynchronous I/O to enrich logs in real time.

Cache frequent lookups or use side‑inputs.

Advanced Q2. How to detect anomalies in system logs near real time?

Flink + sliding window aggregation + z‑score/outlier detection.

Deploy ML models on Flink or forward enriched logs to an anomaly detection service.

Advanced Q3. How to prevent high‑cardinality issues in Prometheus?

Limit dynamic label usage (e.g., user_id, IP).

Use recording rules and down‑sample at scrape time.

The design delivers a production‑ready observability platform capable of handling petabyte‑scale log streams, supporting real‑time metric dashboards, and providing efficient cross‑layer querying while meeting stringent SLA requirements through a Kafka‑Flink‑Elasticsearch‑S3 architecture.