How iLogtail Simplifies Container Log Collection Across Kubernetes Runtimes

Kubernetes and Container Runtime Overview

Kubernetes has become the de‑facto standard for container orchestration. In its architecture, the container runtime (Docker, containerd, etc.) creates, runs and manages containers, and Kubernetes supports multiple runtimes through a modular, standardized interface, giving developers flexibility to choose the most suitable runtime per workload.

CRI and OCI Basics

The Container Runtime Interface (CRI) defines the set of gRPC calls Kubernetes uses to interact with a runtime. OCI (Open Container Initiative) provides specifications for runtime behavior, image format and distribution, forming the foundation for interoperable container runtimes.

Docker

Docker originated in 2010 and uses Linux namespaces and cgroups to provide lightweight containers. Early Kubernetes only supported Docker; when CRI was introduced, Docker required a shim (dockershim) to translate CRI calls to Docker’s native API.

When Kubelet creates a container with Docker, it calls the CRI shim, which forwards the request to the Docker daemon, which in turn delegates to containerd, which finally launches the container via runc.

Containerd

Containerd directly implements CRI, eliminating the need for a separate shim. In version 1.0 it used a dedicated CRI‑containerd process; later versions integrated the CRI logic as a plugin inside the main containerd process.

CRI‑O

CRI‑O is a lightweight runtime built specifically for Kubernetes. It implements both CRI and OCI, offering a minimal footprint while maintaining full compatibility with Kubernetes.

OCI Runtime Specification

OCI defines three specifications: runtime‑spec (container lifecycle), image‑spec (image format) and distribution‑spec (registry interactions). Together with CRI, they form the de‑facto standard for container execution in Kubernetes.

Log Collection Challenges in K8s

Kubernetes log collection faces four main difficulties: short‑lived containers that disappear before logs are harvested, distributed pods across many nodes, diverse storage paths (volumes, host paths, etc.), and the need to correlate logs with container metadata (pod name, namespace, labels).

Open‑Source Collectors

Filebeat

Filebeat is typically deployed as a DaemonSet. It mounts host paths such as /var/log and /var/lib/docker/containers into the Filebeat pod, then reads container log files directly.

filebeat.autodiscover:
  providers:
    - type: kubernetes
      templates:
        - condition:
            equals:
              kubernetes.namespace: kube-system
          config:
            - type: container
              paths:
                - /var/log/containers/*-${data.kubernetes.container.id}.log
              exclude_lines: ["^\\s+[\-`('.|_]" ]

Filebeat watches the Kubernetes API server (list‑watch) to discover pods, namespaces and nodes, caches this information, and enriches logs with labels and annotations.

Fluent Bit

Fluent Bit also runs as a DaemonSet and enriches logs with pod name, pod ID, container name, container ID, labels and annotations by parsing the log file name and querying the API server for additional metadata.

input-kubernetes.conf: |
    [INPUT]
        Name   tail
        Tag    kube.*
        Path   /var/log/containers/*.log
        Parser docker
        DB     /var/log/flb_kube.db
        Mem_Buf_Limit 5MB
        Skip_Long_Lines On
        Refresh_Interval 10
    [FILTER]
        Name   kubernetes
        Match  kube.*
        Kube_URL https://kubernetes.default.svc:443
        Kube_CA_File /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token
        Merge_Log On
        Merge_Log_Key log_processed
        K8S-Logging.Parser On

Fluent Bit can also exclude logs from specific pods using the fluentbit.io/exclude annotation.

iLogtail Solutions

DaemonSet Mode

iLogtail’s DaemonSet mounts the node’s /var/run (for the runtime socket) and the host root filesystem, allowing direct interaction with Docker, containerd or CRI‑O to obtain container metadata and log paths without overloading the Kubernetes API server.

volumeMounts:
  - mountPath: /var/run # container runtime socket
    name: run
  - mountPath: /logtail_host # host log access
    name: root
    readOnly: true
    mountPropagation: HostToContainer

Key benefits include flexible filtering, metadata enrichment (pod name, namespace, node, container status), support for both standard output and custom file logs, multiple mount options, host‑file collection, and reduced API‑server pressure.

Sidecar Mode

In Sidecar mode each pod runs an iLogtail container alongside the application container. The sidecar shares a volume with the application to read custom log files, while also tail‑ing the standard output logs. This mode isolates resource usage per pod and works in serverless or secure‑runtime environments where DaemonSet access is restricted.

apiVersion: v1
kind: Pod
metadata:
  name: counter
spec:
  containers:
  - name: count
    image: busybox
    args: ["/bin/sh","-c","i=0; while true; do echo \"$i: $(date)\" >> /var/log/1.log; echo \"$(date) INFO $i\" >> /var/log/2.log; i=$((i+1)); sleep 1; done"]
    volumeMounts:
    - name: varlog
      mountPath: /var/log
  - name: tail-1
    image: busybox
    args: ["/bin/sh","-c","tail -n+1 -F /var/log/1.log"]
    volumeMounts:
    - name: varlog
      mountPath: /var/log
  - name: tail-2
    image: busybox
    args: ["/bin/sh","-c","tail -n+1 -F /var/log/2.log"]
    volumeMounts:
    - name: varlog
      mountPath: /var/log
  volumes:
  - name: varlog
    emptyDir: {}

Comparison and Advantages

Both DaemonSet and Sidecar modes are supported by open‑source collectors, but iLogtail distinguishes itself by interacting directly with the container runtime, which provides real‑time rootfs information, reduces API‑server load, and works in non‑Kubernetes environments. The sidecar integration also enables deep metadata enrichment for serverless and secure‑runtime scenarios.