Simplify Scalable Kubernetes Pod Logging with Grafana podLogs

01 Why Pod‑level Logs Matter

When a Kubernetes workload misbehaves, developers first inspect pod logs to diagnose failures such as init‑container errors or CrashLoopBackOff, because metrics alone often cannot explain these issues.

Real‑time insight across the whole cluster workload.

Powerful debugging for startup failures, runtime exceptions, and crash loops.

Correlated troubleshooting by joining logs with metrics and Kubernetes events.

Contextual investigation to see what each service was doing at critical moments.

Audit and compliance through long‑term log retention.

The key advantage is unified observability without deploying a separate log agent for each application.

02 From Raw Logs to Enriched Observability

By default, Kubernetes writes container logs to /var/log/containers on each node, organized by namespace and pod UID. These files are rotated by the kubelet, but searching them cluster‑wide or filtering by application version is impractical.

/var/log/pods/
├── namespace1_pod1_uid1/
│   ├── container1/0.log
│   └── container2/0.log
└── namespace2_pod2_uid2/
    └── app/0.log

Vector.dev’s kubernetes_logs component discovers logs across the cluster without sidecars, enriches each line with Kubernetes metadata (namespace, pod name, container, labels, annotations, node, cluster), streams them to the alloy-logs pipeline, and forwards them to Grafana Loki, Grafana Cloud, or other back‑ends. It also handles churn gracefully, so pod restarts or node failures do not interrupt observability.

03 What podLogs Adds

Beyond the raw message, podLogs attaches where and who generated the log, producing structured entries such as:

{
  "timestamp": "2024-01-15T14:30:25.123456789Z",
  "namespace": "production",
  "pod": "web-app-7d4b9c8f6-xyz42",
  "container": "app",
  "node_name": "worker-node-1",
  "app_kubernetes_io_name": "web-service",
  "app_kubernetes_io_version": "v2.1.0",
  "message": "Successfully processed user request ID: abc123 in 45ms"
}

This enrichment enables filtering by application ( app="my-app"), environment ( environment="staging"), or node ( node="node-123"), turning a sea of unstructured text into a searchable, labeled dataset.

04 Enabling podLogs

With the Grafana k8s‑Monitoring Helm chart, enable pod‑level log collection by adding a minimal snippet to values.yaml:

podLogs:
  enabled: true

05 Configuration Options

The podLogs feature offers flexible settings to control collection, enrichment, and post‑processing.

Core Settings

Pre‑Scrape Processing determines how logs are discovered before ingestion. The gatherMethod field selects the discovery mechanism (e.g., volumes).

podLogs:
  enabled: true
  gatherMethod: volumes

Namespace Filtering lets you include or exclude namespaces to avoid noisy system logs:

podLogs:
  namespaces:
    include:
      - production
      - staging
      - monitoring
    exclude:
      - kube-system
      - kube-public

Extra Discovery Rules use pod annotations or names to further refine log selection:

podLogs:
  extraDiscoveryRules: |
    - action: keep
      regex: "true"
      source_labels: [__meta_kubernetes_pod_annotation_logs_enabled]
    - action: drop
      regex: "test.*"
      source_labels: [__meta_kubernetes_pod_name]

Example pod annotation required for collection:

apiVersion: v1
kind: Pod
metadata:
  name: my-app
  annotations:
    logs_enabled: "true"
spec:
  containers:
    - name: app
      image: my-app:latest

Post‑Scrape Processing

Define actions after log collection, such as parsing, enrichment, or security filtering.

extraLogProcessingStages – add parsing, field extraction, sampling, or filtering.

secretFilter – mask sensitive values before forwarding.

podLogs:
  extraLogProcessingStages: |
    - json:
        expressions:
          level: level
          message: msg
          timestamp: ts
    - labels:
        level:
    - match:
        selector: '{level="debug"}'
        stages:
          - sampling:
              rate: 0.1
  secretFilter:
    enabled: true
    includeGeneric: true
    partialMask: 4
    replacement: "[REDACTED]"

When using custom fields, omit the meta_kubernetes prefix; reference labels directly (e.g., {example_com_team="platform"}).

Label and Metadata Control

To avoid high cardinality, explicitly list labels to keep and add structured metadata fields:

podLogs:
  labelsToKeep:
    - app.kubernetes.io/name
    - app.kubernetes.io/instance
    - deployment.environment
    - level
  structuredMetadata:
    k8s.pod.name: k8s.pod.name
    k8s.pod.uid: k8s.pod.uid
    k8s.pod.ip: k8s.pod.ip

06 Full values.yaml Example

The complete configuration can be found at the following URL (plain text, not a hyperlink): https://github.com/varunpappu/articles/blob/main/grafana-k8s-monitoring/alloy-logs/pod-logs/values.yaml

07 Advanced Use Case – Custom Log Paths (Vault Audit Logs)

Enterprise environments may need to collect logs from custom locations such as audit logs on persistent volumes. The alloy-logs module’s extraConfig supports this by defining a custom collection pipeline.

alloy-logs:
  enabled: true
  controller:
    volumes:
      extra:
        - name: host-fs
          hostPath:
            path: /var/lib/kubelet/pods
            type: DirectoryOrCreate
  alloy:
    storagePath: /var/lib/alloy
    mounts:
      extra:
        - name: host-fs
          mountPath: /hostfs/var/lib/kubelet/pods
  extraConfig: |
    declare "audit_logs" {
      argument "audit_logs_destinations" {
        comment = "Must be a list of log destinations where collected logs should be forwarded to"
      }
      discovery.kubernetes "vault_pods" {
        role = "pod"
        namespaces { names = ["vault-enterprise"] }
      }
      discovery.relabel "filtered_vault_pods" {
        targets = discovery.kubernetes.vault_pods.targets
        rule { target_label = "job" replacement = "vault-enterprise/hcvault" action = "replace" }
        rule { action = "labelmap" regex = "__meta_kubernetes_pod_label_(.+)" }
        rule { action = "labelmap" regex = "__meta_kubernetes_pod_annotation_(.+)" }
      }
      discovery.relabel "vault_paths" {
        targets = discovery.relabel.filtered_vault_pods.output
        rule {
          source_labels = ["__meta_kubernetes_pod_uid"]
          target_label = "__path__"
          replacement = "/hostfs/var/lib/kubelet/pods/$1/volumes/kubernetes.io~csi/**/mount/*audit*.log"
        }
      }
      local.file_match "vault_audit" { path_targets = discovery.relabel.vault_paths.output }
      loki.source.file "vault_logs" { targets = local.file_match.vault_audit.targets forward_to = [loki.process.vault_logs.receiver] }
      loki.process "vault_logs" {
        stage.structured_metadata { values = { "k8s_pod_name" = "k8s_pod_name", "k8s_pod_uid" = "k8s_pod_uid" } }
        stage.label_keep { values = ["app_kubernetes_io_name"] }
        forward_to = argument.audit_logs_destinations.value
      }
      audit_logs "feature" { audit_logs_destinations = [loki.write.grafana_cloud_logs.receiver] }
    }

08 Summary

Unified visibility across all namespaces and nodes.

Fast, correlated troubleshooting by joining logs with metrics and events.

Structured, compliant logs suitable for audit and retention.

In short, pod‑level logs become a clear, real‑time narrative of your cluster’s behavior rather than an overwhelming sea of text.