Kubernetes Observability Blueprint: How to Implement Monitoring, Logging, and Tracing

Why Tracing Matters

A pod restarts, CPU and memory look normal, but the container exits immediately; logs show no panic or OOM. The missing piece is tracing —knowing which services the pod communicated with before failure. This illustrates the purpose of the three observability pillars.

1. Observability ≠ Monitoring

Monitoring is "knowing what questions to ask and setting checkpoints in advance". Observability is "inferring internal state from external behavior after a problem occurs". In dynamic, distributed Kubernetes environments, traditional monitoring hits a ceiling quickly, while the three pillars let you pinpoint issues within seconds.

Pillars and their roles:

Metrics (e.g., Prometheus): answer "when, which service, what value"; store time‑series data.

Logs (e.g., Loki or ELK): answer "what happened, with what context"; store unstructured text events.

Traces (e.g., Tempo or Jaeger): answer "which nodes were traversed and how long each step took"; store causal trees.

All three are linked by a common TraceID that creates a unified observability system.

2. Metrics – Prometheus Ecosystem

Metrics are the foundation. Instead of assembling Prometheus, Grafana, Alertmanager, and Node Exporter manually, use the kube-prometheus-stack Helm chart:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --create-namespace \
  --set prometheus.prometheusSpec.retention=30d \
  --set prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.storageClassName=fast-ssd \
  --set prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.resources.requests.storage=200Gi

After installation you get:

Prometheus – metric collection and storage

Grafana – dashboards (20+ K8s panels)

Alertmanager – alert routing

Node Exporter – node‑level CPU/memory/disk/network

kube‑state‑metrics – K8s object metrics

Prometheus Operator – CRD‑based config management

Google SRE’s four “golden metrics” guide metric selection:

Latency –

histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))

Traffic – sum(rate(http_requests_total[5m])) by (service) Errors –

sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))

Saturation –

container_cpu_usage_seconds_total / kube_node_status_allocatable

Recommended monitoring methods: USE (Utilization + Saturation + Errors) for nodes, disks, networks; RED (Rate + Errors + Duration) for services.

Declarative monitoring with ServiceMonitor CRD:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: payment-api
  namespace: monitoring
  labels:
    release: kube-prometheus-stack
spec:
  selector:
    matchLabels:
      app: payment-api
  namespaceSelector:
    matchNames:
    - production
  endpoints:
  - port: metrics
    interval: 15s
    path: /metrics
    relabelings:
    - sourceLabels: [__meta_kubernetes_pod_node_name]
      targetLabel: node

ServiceMonitor lets teams declare monitoring targets in their own namespace, aligning with GitOps.

3. Logs – Loki vs. ELK

ELK’s three pain points in K8s:

High storage cost due to full‑text indexing.

Operational complexity (cluster tuning, sharding, replicas).

Additional agents (Filebeat/Fluentd) for K8s integration.

Loki solves them by indexing only labels , reducing storage by an order of magnitude, providing a single binary, and offering native K8s support via Promtail .

Deploy Loki stack with Helm:

helm repo add grafana https://grafana.github.io/helm-charts
helm install loki-stack grafana/loki-stack \
  --namespace monitoring \
  --set loki.persistence.enabled=true \
  --set loki.persistence.size=100Gi \
  --set promtail.enabled=true \
  --set promtail.config.snippets.pipelineStages[0].docker={}

Promtail runs as a DaemonSet, harvesting logs from /var/log/pods/. Example pipeline to extract JSON fields and promote trace_id to a label:

scrape_configs:
- job_name: kubernetes-pods
  kubernetes_sd_configs:
  - role: pod
  pipeline_stages:
  - json:
      expressions:
        level: level
        trace_id: trace_id
        msg: message
  - labels:
      level:
      trace_id:
  - timestamp:
      source: timestamp
      format: RFC3339Nano

Ensuring trace_id is extracted as a label is the key step for cross‑pillar correlation.

4. Tracing – Tempo + OpenTelemetry

Traditional tracing requires manual SDK instrumentation in every service. The OpenTelemetry Operator injects the SDK automatically via an Admission Webhook.

# Install OTel Operator
kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml

# Deploy Tempo (trace storage)
helm install tempo grafana/tempo \
  --namespace monitoring \
  --set persistence.enabled=true \
  --set persistence.size=50Gi

Enable auto‑injection by labeling the namespace:

kubectl label namespace production instrumentation.opentelemetry.io/inject-java="true"

Define an Instrumentation resource:

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: my-instrumentation
  namespace: production
spec:
  exporter:
    endpoint: http://tempo.monitoring.svc:4317
  propagators: ["tracecontext", "baggage"]
  sampler:
    type: parentbased_traceidratio
    argument: "0.1"
  java:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:latest
  python:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:latest

Result: newly deployed Java/Python pods automatically receive the OTel agent and send traces to Tempo without code changes.

Example trace tree (simplified):

Trace (trace_id: abc123)
├── Span 1: HTTP GET /checkout (service: gateway, 850ms)
│   ├── Span 2: HTTP POST /payment (service: payment‑api, 620ms)
│   │   ├── Span 3: DB Query (service: payment‑api, 180ms)
│   │   └── Span 4: HTTP POST /risk (service: risk‑api, 380ms)
│   │       └── Span 5: Redis GET (service: risk‑api, 45ms)
│   └── Span 6: HTTP GET /inventory (service: inventory‑api, 180ms)

This causal view shows the bottleneck (payment‑api) that metrics or logs alone cannot reveal.

5. Correlating the Three Pillars

When the pillars operate in isolation they lose half their value. The unified approach uses TraceID to jump between them:

Metrics panel shows latency spike → click Exemplar → open Trace → copy TraceID → filter Logs by TraceID → see full error stack

Enable Prometheus Exemplar (available from v2.26):

# prometheus.yaml
prometheus:
  prometheusSpec:
    enableFeatures:
    - exemplar-storage
    retentionExemplars: 7d

Attach TraceID to a metric:

# original metric
http_request_duration_seconds_bucket{le="0.5",service="payment"} 1234

# with exemplar
http_request_duration_seconds_bucket{le="0.5",service="payment"} 1234 # {trace_id="abc123"} 0.42

Grafana configuration to link Trace → Logs (Tempo datasource with Loki as tracesToLogs):

tracesToLogs:
  datasourceUid: loki-datasource
  tags: ['service.name', 'trace_id']
  mappedTags:
    'service.name': 'service'
  filterByTraceID: true
  filterBySpanID: false

Clicking a trace now jumps directly to the related logs.

6. Unified Grafana Dashboard

Instead of three separate UIs, Grafana can display Metrics, Logs, and Traces together. Data source definitions:

apiVersion: 1
datasources:
- name: Prometheus
  type: prometheus
  url: http://prometheus.monitoring.svc:9090
  isDefault: true
- name: Loki
  type: loki
  url: http://loki.monitoring.svc:3100
  jsonData:
    derivedFields:
    - datasourceUid: tempo
      matcherRegex: 'trace_id=(\w+)'
      name: TraceID
      url: '$${__value.raw}'
- name: Tempo
  type: tempo
  url: http://tempo.monitoring.svc:3200
  jsonData:
    tracesToLogs:
      datasourceUid: loki
      filterByTraceID: true

Dashboard layering (four levels):

Level 1 – Global health : SLO burn rate, error budget (Prometheus).

Level 2 – Service view : RED metrics (Rate, Errors, Duration) (Prometheus).

Level 3 – Infrastructure : Pod status, node resources (Prometheus + kube‑state‑metrics).

Level 4 – Drill‑down : Specific logs and traces (Loki + Tempo).

Typical troubleshooting flow: start at level 1, drill to level 2 for latency, then to level 3 for node issues, finally to level 4 for logs and trace details.

7. eBPF – The New Observability Paradigm

eBPF runs sandboxed programs in the Linux kernel, allowing you to capture syscalls, network packets, and more without modifying application code.

No need for /metrics endpoints – eBPF collects CPU/memory/network directly.

No SDK required – eBPF intercepts syscalls and builds traces.

No pod changes – eBPF runs on the node.

Cilium Hubble (network observability) example:

# Enable Hubble
helm upgrade cilium cilium/cilium \
  --namespace kube-system \
  --reuse-values \
  --set hubble.enabled=true \
  --set hubble.metrics.enabled="{dns,drop,tcp,flow,icmp,http}" \
  --set hubble.ui.enabled=true

# Observe real‑time traffic
hubble observe --pod production/payment-api --protocol http

Hubble reveals things traditional monitoring cannot, such as pod‑to‑pod connection refusals, DNS latency, HTTP method/path/status, and TCP retransmission rates.

Pixie provides auto‑tracing via eBPF:

# One‑click install
helm install pixie pixie-operator/pixie-operator-chart \
  --namespace pl \
  --create-namespace

# View HTTP requests
px run http_data --cluster production

Pixie captures HTTP/gRPC traffic, builds traces and spans without any OTel SDK or code changes, making it a rescue tool for legacy systems.

8. Selection Decision Matrix

Tooling choices differ by team size and maturity:

Small (<50 Pods) : single‑node Prometheus + Loki; no tracing.

Medium (50‑500 Pods) : full stack – kube‑prometheus‑stack, Loki + Promtail, Tempo + OTel.

Large (500‑5000 Pods) : replace Prometheus with a VM‑backed store, scale Loki and Tempo clusters.

Very large (>5000 Pods) : VM cluster or Mimir for metrics, micro‑service Loki, federated Tempo.

Key selection principles:

Start with Metrics, then Logs, finally Traces (Metrics give the best ROI).

Prefer Loki over ELK unless full‑text search is mandatory.

Invest in OpenTelemetry as the long‑term standard; Jaeger/Zipkin/SkyWalking are converging toward OTel.

Adopt eBPF only after the three pillars are solid; it’s an enhancement, not a replacement.

9. Pitfall Guide

Pitfall 1 – Prometheus high‑cardinality OOM

Symptom: memory usage climbs until OOM.

Cause: using high‑cardinality labels such as user_id or request_id creates a separate time series per value.

# Wrong – user_id as label
http_requests_total{user_id="12345"} 1
http_requests_total{user_id="12346"} 1
# 1 M users → 1 M series

# Correct – keep user_id out of labels
http_requests_total{method="GET",status="200"} 1000000

Solution: monitor series count with prometheus_tsdb_head_series and avoid high‑cardinality labels; use Exemplar or logs for such identifiers.

Pitfall 2 – Loki query timeout

Symptom: 24‑hour log query returns 504.

Cause: Loki scans many blocks because no label filter is applied.

Solution: always include namespace or app label to narrow the range.

# Slow (full scan)
{job="kube-system/kubelet"}

# Fast (label filter)
{namespace="production", app="payment-api"} |= "error"

Pitfall 3 – OTel sampling too high

Symptom: Tempo storage explodes, queries become slow.

Cause: collecting every trace (tens of thousands per second).

Solution: use tail‑based sampling in the OTel Collector.

processors:
  tail_sampling:
    decision_wait: 30s
    policies:
    - name: errors
      type: status_code
      status_code:
        status_codes: [ERROR]
    - name: slow
      type: latency
      latency:
        threshold_ms: 500
    - name: baseline
      type: probabilistic
      probabilistic:
        sampling_percentage: 10

Effect: 100 % of error and slow traces kept, 10 % of the rest – storage drops ~80 %.

Pitfall 4 – Grafana dashboard overload

Symptom: opening a dashboard freezes or crashes the browser.

Cause: >30 panels each issuing multiple queries.

Solution: limit a dashboard to ≤12 panels and split into multiple dashboards by layer.

Pitfall 5 – Promtail missing pod labels

Symptom: logs for some pods are absent.

Cause: Promtail starts after the pod crashes, so metadata isn’t captured.

Solution: add relabel_configs to preserve full K8s metadata.

relabel_configs:
- source_labels: [__meta_kubernetes_pod_name]
  target_label: pod
- source_labels: [__meta_kubernetes_namespace]
  target_label: namespace
- source_labels: [__meta_kubernetes_pod_label_app]
  target_label: app

Pitfall 6 – TraceID mismatch between Logs and Tempo

Symptom: Grafana trace‑to‑log link returns no results.

Cause: different formatting (e.g., one has 0x prefix).

Solution: normalize TraceID in Promtail pipeline.

- regex:
    expression: 'trace_id=(\w+)'
    source: message
- template:
    source: trace_id
    template: 'Lower(.TrimPrefix(Value "0x"))'
- labels:
    trace_id:

10. Implementation Roadmap

Phase 1 – Metrics First (Weeks 1‑2)

Goal: establish infrastructure and service‑level monitoring.

Deploy kube-prometheus-stack.

Expose metrics for all nodes and core services.

Create three Grafana dashboards: cluster overview, node resources, service RED.

Configure basic Alertmanager rules.

Acceptance: Grafana shows CPU/memory per pod and QPS/latency per service.

Phase 2 – Log Ingestion (Weeks 3‑4)

Goal: collect and query all application logs.

Deploy Loki + Promtail.

Ensure Promtail extracts namespace, pod, app labels.

Log query latency < 3 s for a 24‑hour range.

Standardize log format as JSON with a trace_id field.

Acceptance: Grafana can retrieve recent logs for any service using namespace + app filters.

Phase 3 – Tracing & Correlation (Weeks 5‑8)

Goal: enable tracing and close the three‑pillar loop.

Deploy Tempo.

Configure OTel Operator for automatic injection.

Enable Prometheus Exemplar support.

Set up Grafana trace‑to‑log linking.

Provide an end‑to‑end troubleshooting dashboard.

Acceptance: From a Grafana metric panel → click Exemplar → open Trace → jump to related logs in ≤5 clicks.

Conclusion

The three pillars are the starting point, not the finish line. The next step is adding the fourth pillar—profiling (e.g., Pyroscope or Parca) for CPU/memory hot‑spot analysis. Ultimately, eBPF will collect all observability data from the kernel, but the core principle remains: shorten MTTR, not accumulate tools.