Mastering Kubernetes Jobs, CronJobs, and DaemonSets: Concepts, YAML, and Real‑World Walkthroughs

Background : In Kubernetes the smallest scheduling unit is a Pod, but using Pods directly for batch workloads raises issues such as ensuring proper termination, retrying failures, handling dependencies, and controlling parallelism. The Job controller, its sibling CronJob , and the DaemonSet controller address these problems.

Job Controller

A Job creates one or more Pods to run a task to completion. It tracks success/failure, supports restart policies ( Never, OnFailure, Always) and a retry limit via backoffLimit. Example YAML (simplified):

apiVersion: batch/v1
kind: Job
metadata:
  name: pi
spec:
  template:
    spec:
      containers:
      - name: pi
        image: perl
        command: ["perl", "-Mbignum=bpi", "-wle", "print bpi(2000)"]
      restartPolicy: Never
  backoffLimit: 4

Key status fields displayed by kubectl get jobs include AGE (how long the Job has existed), DURATION (runtime of the task), and COMPLETIONS (how many Pods finished).

Parallel Jobs

Parallelism is controlled by two spec fields:

completions : total number of successful Pod runs required.

parallelism : maximum number of Pods that may run simultaneously (the “queue size”).

For example, completions: 8 and parallelism: 2 cause the Job to run eight Pods in four batches of two, which can be observed with kubectl get pods and the pod‑age column.

CronJob Controller

CronJob extends Job with a schedule expressed in standard Linux cron syntax. Important fields:

schedule : e.g., */1 runs the job every minute.

startingDeadlineSeconds : maximum time to wait for a missed schedule before giving up.

concurrencyPolicy : Allow (run overlapping jobs) or Forbid (wait for the previous run to finish).

successfulJobsHistoryLimit / failedJobsHistoryLimit : how many past Job objects to retain.

A typical CronJob YAML looks like:

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: hello
spec:
  schedule: "*/1"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: hello
            image: busybox
            args: ["/bin/sh", "-c", "date; echo Hello from the Kubernetes cluster"]
          restartPolicy: OnFailure

After creation, kubectl get cronjobs shows the schedule, and the generated Jobs appear under kubectl get jobs each minute.

DaemonSet Controller

DaemonSet ensures that a copy of a Pod runs on every (or a selected subset of) node. Typical use‑cases include log collectors, storage agents, and node‑level monitoring. Core capabilities:

Deploy one Pod per node automatically.

When a new node joins, DaemonSet creates the corresponding Pod.

When a node leaves, its Pod is removed.

Tracks pod health and recreates crashed Pods.

Example DaemonSet YAML (Fluentd collector):

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd
spec:
  selector:
    matchLabels:
      name: fluentd
  template:
    metadata:
      labels:
        name: fluentd
    spec:
      containers:
      - name: fluentd
        image: fluent/fluentd
        resources:
          limits:
            memory: 200Mi

After applying, kubectl get daemonset reports desired, current, and ready pod counts. nodeSelector can limit DaemonSet to specific node groups (e.g., only master nodes).

Update Strategies

Both Deployment‑style controllers (DaemonSet, Job) support two update policies:

RollingUpdate : Pods are updated one by one, ensuring a smooth transition.

OnDelete : New Pods are created only after the old ones are manually deleted, giving full manual control.

Observing the DaemonSet status after kubectl apply -f daemonset.yaml shows which Pods have been updated (e.g., 0‑4 → 1‑4).

Architecture Overview

Both Job and DaemonSet controllers watch the API server for changes to their custom resources and to Nodes. When a discrepancy is detected—missing Pods, node additions, or configuration changes—the controller enqueues a work item, creates or deletes Pods, and finally writes the updated status back to the API server.

Summary

Jobs and CronJobs provide reliable batch and scheduled execution with configurable retry and parallelism.

DaemonSets guarantee per‑node pod deployment, useful for agents, log collectors, and monitoring.

Understanding the YAML fields and controller workflows enables precise control over workload lifecycle in a Kubernetes cluster.