Building Lightweight Log Systems with Loki and Alloy: From Deployment to Query Optimization

Why Loki Over ELK

For medium-scale systems (20-50 nodes, 30-100 GB/day), ELK is often too heavy: it requires a dedicated Elasticsearch cluster (3+ data nodes, 16C64G+SSD each), JVM tuning, shard planning, and hot/cold tiering. Yet 95% of logs are never queried after writing, while ES still maintains inverted indexes, shards, and heap memory for them. Loki's design trade-off — index only labels, not log content — stores logs as labeled text chunks in object storage, reducing storage to 1/5-1/10 of ES. The cost: full-text scans are slower and scale with scanned data volume.

A real-world comparison (60 GB/day, 30-day retention): ELK used ~2.4 TB (3×16C64G+2 TB SSD) with 2 s full-text P95; Loki used ~220 GB (1×8C32G + object storage) with 3-8 s label+filter P95, and 0.5-1.5 s with bloom filters for high-cardinality filters. The key insight: Loki succeeds when queries are "I know which service/host/time range, find errors"; it fails for multi-dimensional aggregations like "P99 latency per endpoint per day" — use metrics or ClickHouse for that.

Promtail EOL and Migration to Alloy

Promtail reached EOL on 2026-03-02 and was removed from the Loki repo in v3.7.3. Existing deployments can run but receive no security patches. New deployments should use Grafana Alloy, which also collects metrics and traces. An official migrator converts promtail config.yml to config.alloy. Migration risks: positions file incompatibility (causes re-reading entire files), regex escaping differences (YAML \d → Alloy \\d), and __path__ requiring absolute paths. Dual-run validation: compare line counts per collector label over 1h windows; investigate >2% differences.

Core Data Model: Streams, Chunks, Index

Log line → Stream (unique label set, e.g., {job="nginx", instance="10.0.10.21", env="prod"}) → Chunk (compressed ~1.5 MB or 1h, controlled by chunk_target_size, max_chunk_age) → Object storage. Index only maps label sets to chunks; TSDB index lives in object storage with local cache. Query {job="nginx"} |= "error" executes: 1) index lookup for matching streams, 2) fetch chunks from object storage, 3) substring filter on chunk text, 4) return matching lines. Step 2 is the bottleneck — matching 1000 streams × 720 chunks = 720k chunk fetches. Hence label cardinality must be controlled.

Label Cardinality: The Lifeline

Cardinality = number of distinct label values. {env="prod"} = 1; {instance="10.0.10.21"} = 50 in a 50-node cluster; {request_id="a1b2c3"} = request count (millions). High cardinality kills Loki: each unique label set = a stream, each stream holds a head block in ingester memory; exceeding max_global_streams_per_user (default 5000) rejects writes; more streams = more chunks to scan = slower queries; index size grows linearly. Rules: keep active streams <100k per tenant; label cardinality product <100k; never use high-cardinality fields (user_id, request_id, trace_id, path, IP) as labels — use structured metadata (Loki 3.x) or filter in log line.

Wrong: {job="nginx", path="/api/v1/orders/12345"} (thousands of paths). Right: {job="nginx"} |= "/api/v1/orders/" or structured metadata {job="nginx"} | path="/api/v1/orders/12345" (requires Loki 3.x + collector support).

Four Critical Config Knobs

limits_config.ingestion_rate_mb

: per-tenant write rate (MB/s). Too low → 429s, log loss; too high → single tenant overwhelms ingester. limits_config.max_global_streams_per_user: stream count limit. Too low → label mistakes reject writes. limits_config.max_query_series: max streams per query. Too large → queries can drag down cluster. limits_config.max_entries_limit_per_query: max returned lines. Too small → incomplete results; too large → browser freeze.

All four relate directly to label design. Good labels make them rarely trigger; bad labels make any value insufficient.

Collector Pipeline (Promtail/Alloy)

Three stages: discover files → attach labels → push. Promtail pipeline stages run serially: docker/cri (parse container logs) → multiline (merge Java stack traces) → json/regex (extract fields) → labels (promote fields to labels) → structured_metadata (store high-cardinality fields as metadata) → timestamp (use log's timestamp) → output (rewrite format). Alloy uses components connected via forward_to with same semantics. Common mistake: putting high-cardinality fields in labels stage — Promtail creates hundreds of thousands of streams, Loki then rejects writes. Label design must be finalized before writing config.

Deployment: Single Binary + Object Storage

Most teams should start with single-binary mode (all components in one process) backed by S3/MinIO. A single 8C32G node handles 30-50 MB/s write (~3-4 TB/day), covering 90% of workloads. Scale out only when needed. Capacity estimation: daily ingest = lines/sec × avg line bytes × 86400; storage ≈ daily ingest × compression ratio (0.1) × retention days × replication factor. Example: 5000 lines/s × 300 bytes × 86400 ≈ 130 GB/day raw; 130 GB × 0.1 × 14 days ≈ 182 GB. Memory: ~2-4 GB per 100k active streams; 8C32G safe up to ~300k streams.

Loki Config Highlights (v3.7.x, schema v13 + TSDB)

schema_config.from

must be a past date (e.g., 2024-04-01); future date causes no schema config found for date on startup. period: 24h is mandatory for TSDB; other values fail.

Use ${ENV_VAR} for S3 credentials; never hardcode. Enable -config.expand-env=true if needed. retention_period requires compactor.retention_enabled: true; otherwise logs never delete. delete_request_store must match object_store (e.g., both s3).

Set max_global_streams_per_user to a finite value (e.g., 20000) as a guardrail.

Verification Steps

curl /ready

→ "ready" (initial "Ingester not ready: waiting for 15s" is normal).

curl /config | grep retention_period|max_global_streams|auth_enabled

confirms config loaded.

Push test log with nanosecond timestamp: $(date +%s)000000000; expect HTTP 204.

Query back with query_range; if 204 but no results, check timestamp unit (seconds vs nanoseconds) and query time range.

LogQL: From Working to Fast

Query = stream selector + filter chain + aggregation. Selector must have at least one non-empty matcher; {job=~".*"} is rejected. Good: {job="nginx", env="prod"}, {job=~"nginx.*"} (prefix regex allows index pruning). Bad: {env="prod"} (too many streams), {job=~".*nginx"} (suffix regex = full scan). Filters: |= "str" (substring, fast), |~ "re" (regex, slow), | json | key="v" (parse JSON then exact field compare, faster than regex on raw line), | pattern "<ip> ..." (structured pattern, faster than regex). Performance tips: 1) coarse filter with |= then fine with |~; 2) prefer | json | field>=500 over regex; 3) pattern beats regex; 4) avoid case-insensitive regex (?i) — it disables literal optimization.

Common templates: service errors {job="app", env="prod"} |= "ERROR"; exclude noise {job="app"} != "/healthz" != "/metrics" |= "ERROR"; JSON field filter {job="app"} | json | level="error" | duration > 3; rate by pod sum by (pod) (count_over_time({job="app"} |= "ERROR" [1m])); extract latency P99

quantile_over_time(0.99, {job="nginx"} | json | unwrap upstream_time [5m]) by (app)

; structured metadata filter {job="app"} | thread="http-nio-8080-exec-3".

Red lines: 30s timeout → narrow selector/time range; maximum of series (50000) reached → narrow query or raise max_query_series; query timed out in expression evaluation → expensive aggregation (unwrap + large range) → reduce range or pre-aggregate; entry limit reached → add filters or check scale first; browser freeze → use metrics panels, not logs panels for large ranges. Habit: run sum(count_over_time({job="app"} |= "ERROR" [1h])) before log query; if millions, don't query raw logs.

Recording Rules & Alerting

Pre-aggregate log-derived metrics via ruler (interval 1m). Rules stored under rules/fake/ for single-tenant ( auth_enabled: false uses tenant "fake"). Example: app:log_errors:rate1m =

sum by (namespace, app) (count_over_time({job="app"} |= "ERROR" [1m]))

. Alert on spikes:

sum by (namespace, app) (count_over_time({job="app"} |= "ERROR" [5m])) > 50

for 5m. Annotations should include "where to look" links.

Monitoring Loki Itself

Scrape /metrics on HTTP port. Key metrics: loki_distributor_lines_received_total (cross-check with collector), loki_distributor_ingester_append_failures_total (non-zero = ingester issues), loki_request_duration_seconds (query latency), loki_ingester_streams_created_total (spike = label design issue), loki_ingester_memory_streams (approaching limit = check labels), loki_compactor_deleted_lines_total (retention working?). Five essential alerts: 1) 429 rate-limiting (>0.5/s for 10m), 2) ingester append failures (>0 for 5m, critical), 3) ingester memory streams >150k for 15m, 4) chunk flush P99 >30s for 10m, 5) query P95 >10s for 10m. Verify metric/label names per version before deploying.

Collector Health

Alloy exposes metrics at --server.http.listen-addr. Alert on loki_write_dropped_bytes_total > 0 (logs being dropped — Loki side shows nothing). Common causes: Loki unreachable, 429 backpressure, undersized queue.

Top 5 Troubleshooting Scenarios

1. Written but not queryable

Push returns 204 but Grafana shows nothing. Check: stream exists ( /label/job/values), query without filters over 24h. Most common: timestamp unit wrong (seconds/milliseconds instead of nanoseconds → lands in 1970), timezone offset (8h shift if location wrong), or reject_old_samples dropping bulk backfill (look for entry too far behind or discarded_samples_total{reason="too_far_behind"}). Fix: correct collector timestamp parsing, don't increase reject_old_samples_max_age for historical imports — use separate label and accept timestamps.

2. Duplicate or missing logs

Duplicates:

count by (instance) (count_over_time({job="app"} |= "unique_string" [1h]))

. Causes: lost positions file (re-read from start), duplicate collection (two tasks reading same file), application double-logging. Missing: check collector loki_write_dropped_bytes_total and Loki discarded_samples_total with reason labels (rate_limited, too_far_behind, out_of_order). Loki requires monotonic timestamps per stream; multi-replica collectors or multi-threaded app writing same file cause out-of-order.

3. Query timeout / "too many outstanding requests"

Check concurrent queries ( loki_querier|inflight), slow query logs, test minimal query ( limit=1 since=1m). If minimal fast but large slow: time range too wide + wide selector, unwrap + large range quantile_over_time, dashboard with many panels querying 7 days simultaneously. Fix: default panel range 1h, use recording rules for dashboards, increase querier.max_concurrent and query_frontend.max_outstanding_per_tenant with memory headroom.

4. Disk growth despite retention

Verify: compactor.retention_enabled: true (most common miss), compactor running ( loki_compactor_runs/deleted metrics), index/chunk sizes. retention_delete_delay (default 2h) delays visible deletion; max_chunk_age affects actual deletable time; object storage versioning/recycle bin adds overhead. Clear local index cache ( tsdb_shipper.cache_location) if it balloons — safe to delete, rebuilds on restart.

5. Grafana empty but curl works

Data source URL uses localhost (Grafana container resolves to itself) → use service name/IP. Panel time range vs query range mismatch. Template variable $app empty → {job=""} returns nothing; set default values. Old Grafana Loki plugin lacks structured metadata syntax support → check browser console / Grafana logs.

Multi-tenancy & Security

Enable auth_enabled: true only for strict team isolation. Requires X-Scope-OrgID header on all requests; collector sets tenant_id; limits/retention per tenant. Switching on after single-tenant use makes historical data (written under "fake") inaccessible — decide early. Loki does no user auth; auth_enabled only isolates tenants. Never expose 3100 to untrusted networks; use NetworkPolicy/firewall. For auth in front, use Nginx Basic Auth or OAuth2 Proxy injecting X-Scope-OrgID. Logs centralize sensitive data (PII, tokens, DB passwords). Collector-side redaction (Promtail replace, Alloy stage.replace) e.g., phone regex (1[3-9]\d{9}) → ***. Limits: CPU overhead at high throughput; not a compliance guarantee (misses variants); over-redaction breaks debugging (payment tracing needs full IDs). Best fix: application stops logging secrets.

Performance Tuning Knobs

chunk_target_size

(1.5 MB default): larger → fewer object PUTs, more memory, slower chunk fetch. max_chunk_age (2h): larger → better compression, more data loss on ingester crash. chunk_encoding: snappy (fast, default) vs gzip (better ratio, more CPU). Test in staging. chunk_idle_period (30m): smaller → more object requests; larger → more memory. query_range.align_queries_with_step: true boosts result cache hit rate (default off). results_cache TTL 24h good for historical, but may stale "last 5m" panels — verify by writing test log and checking 30s refresh.

Single-node limits (8C32G, SSD, filesystem): write 30-50 MB/s, active streams 200-300k, concurrent queries 10-20. Bottlenecks: CPU for write, memory for streams, CPU for queries. Adding CPU cores doesn't fix stream-count memory pressure.

MinIO Backend Gotchas

s3forcepathstyle: true

required (MinIO doesn't support virtual-hosted-style without DNS). insecure must match protocol (http→true, https→false).

MinIO IOPS directly impacts Loki write latency; use SSD or erasure-coded multi-disk.

Summary Checklist

Promtail EOL → new deployments use Alloy; migrate existing with dual-run validation.

Loki lives/dies by label design, not deploy params. Keep cardinality low; use structured metadata for high-cardinality fields.

Three config guardrails: schema_config.from (past date), compactor.retention_enabled: true, max_global_streams_per_user (finite).

Always estimate result size with count_over_time before raw log query.

Monitor Loki itself: 429s, ingester failures, memory streams, chunk flush latency, collector dropped bytes.

Secure by default: no auth on Loki, protect 3100, redact at collector/app layer.

Choose Loki for "find errors in service X at time Y"; choose ES/ClickHouse for "aggregate/sort/filter arbitrarily". Wrong tool choice cannot be parameter-tuned away.