Mastering Kubernetes DaemonSets: Ensure One Pod per Node Efficiently

What is a DaemonSet?

As the name suggests, a DaemonSet ensures that a daemon pod runs on every (or selected) node in a Kubernetes cluster. When nodes are added, pods are created; when nodes are removed, the pods are garbage‑collected. Deleting the DaemonSet removes its pods.

Characteristics of daemon pods

Runs on each node (usually).

Only one such pod per node.

Automatically created on newly added nodes.

Automatically removed when a node is deleted.

Typical uses

Cluster storage daemons (e.g., ceph).

Log collection daemons (e.g., fluentd).

Node monitoring daemons (e.g., Prometheus node exporter).

Kubernetes system daemonsets

Kubernetes itself runs system components as DaemonSets. For example:

$ kubectl get daemonset -n kube-system
NAME        DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
aws-node    2        2       2      2          2         <none>        160d
kube-proxy  2        2       2      2          2         <none>        160d

These are the aws-node and kube-proxy DaemonSets.

DaemonSet definition

A typical DaemonSet YAML looks like:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd-elasticsearch
  namespace: kube-system
  labels:
    k8s-app: fluentd-logging
spec:
  selector:
    matchLabels:
      name: fluentd-elasticsearch
  template:
    metadata:
      labels:
        name: fluentd-elasticsearch
    spec:
      tolerations:
        # allow running on master nodes
        - key: node-role.kubernetes.io/master
          operator: Exists
          effect: NoSchedule
      containers:
      - name: fluentd-elasticsearch
        image: quay.io/fluentd_elasticsearch/fluentd:v2.5.2
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 200Mi
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
      terminationGracePeriodSeconds: 30
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers

This DaemonSet runs a fluentd-elasticsearch pod that forwards container logs to Elasticsearch via fluentd. To avoid excessive host resource usage, add a resources field to limit CPU and memory.

How DaemonSet ensures one pod per node

The DaemonSet controller fetches the node list from the API server, iterates each node, and checks for existing DaemonSet pods. Three outcomes are possible: no pod (create one), more than one pod (delete extras), exactly one pod (node is healthy).

Node selection

To schedule pods on specific nodes, set .spec.template.spec.nodeSelector. If omitted, pods are created on all nodes. Modern clusters prefer nodeAffinity over the deprecated nodeSelector:

apiVersion: v1
kind: Pod
metadata:
  name: with-node-affinity
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: metadata.name
            operator: In
            values:
            - node-name

Tolerations

DaemonSets automatically add a toleration so that pods can run on nodes tainted as unschedulable:

apiVersion: v1
kind: Pod
metadata:
  name: with-toleration
spec:
  tolerations:
  - key: node.kubernetes.io/unschedulable
    operator: Exists
    effect: NoSchedule

This allows the DaemonSet pod to ignore the NoSchedule taint and be placed on every node.

Creating the fluentd DaemonSet

Apply the YAML with:

$ kubectl create -f fluentd-elasticsearch.yaml

Verify creation:

$ kubectl get pod -n kube-system -l name=fluentd-elasticsearch
NAME                         READY STATUS  RESTARTS AGE
fluentd-elasticsearch-dqfv9 1/1   Running 0        10m
fluentd-elasticsearch-pf9z5 1/1   Running 0        10m

List the DaemonSet:

$ kubectl get ds -n kube-system fluentd-elasticsearch
NAME                DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
fluentd-elasticsearch 2      2       2     2          2         <none>        1h

Conclusion

A DaemonSet manages only pod objects, using nodeAffinity and tolerations to guarantee a single pod per node. Its controller works similarly to a Deployment, making it straightforward to understand.