How to Build Production‑Grade Observability Metrics and Alerting for Batch Jobs

Why Batch Jobs Fail Silently

Online services expose failures quickly, but batch jobs run unattended for minutes to hours, so problems are discovered only after business impact. A typical incident shows a nightly settlement job that appears healthy in Kafka lag and CPU usage, yet a single shard slows from 200 ms to 18 s, causing downstream backlog.

Tasks rarely fail completely; they become partially slow.

Partial failures are hidden by retries.

Logs lack a unified context (run_id / shard_id / checkpoint).

System metrics exist, but no task‑level health portrait.

Observability means turning a long‑running, multi‑component task with retries and compensation into a quantifiable, correlatable, alertable, traceable execution picture.

What Constitutes Batch Observability

Many teams only monitor success/failure counts, error logs, and a timeout alarm – this is the surface layer. Production‑grade observability must cover four layers:

1. Task Status Visibility

Is the task started, running, completed, failed, or partially completed?

Which execution round is it?

Which tenant, business domain, or data window does it belong to?

2. Task Progress Visibility

Total work amount.

How many records have been processed.

Remaining work.

Estimated completion time.

Which shard/partition/batch is the bottleneck.

3. Task Quality Visibility

Success, failure, skip, and retry rates.

Top error categories.

Presence of dirty data, idempotent conflicts, downstream write conflicts, dead‑letter accumulation.

4. Task Performance & Resource Visibility

Per‑record, batch, and end‑to‑end latency.

CPU, memory, connection pool, coroutine pool, thread pool, GC usage per task instance.

Whether slowdown is due to business logic, resource contention, or downstream dependency.

Modeling the Task State Machine

PENDING
 -> DISPATCHED
 -> RUNNING
 -> PARTIALLY_SUCCEEDED
 -> SUCCEEDED
 -> FAILED
 -> CANCELLED
 -> TIMEOUT

Key distinctions: FAILED: whole execution failed, result unusable. PARTIALLY_SUCCEEDED: some shards/records failed, main flow finished, compensation needed. TIMEOUT: task did not fail outright but exceeded acceptable window.

Batch Event Model

TaskCreated
TaskStarted
ShardAssigned
BatchFetched
RecordProcessed
RecordFailed
RecordRetried
CheckpointCommitted
ShardCompleted
TaskCompleted
TaskFailed
TaskTimedOut

With a unified event flow, metrics, logs and traces can be derived automatically.

Layered Architecture for Production Observability

4.1 Layered Stack

Scheduler/Workflow (Airflow, XXL‑Job, Argo, CronJob) → Batch Worker Cluster → Task SDK (metrics, logs, traces, context) → Prometheus / Pushgateway or OpenTelemetry Collector → Loki / Elasticsearch → Jaeger / Tempo / SkyWalking → MySQL / Kafka / Redis / ES / Object Storage → Grafana Dashboard → Alertmanager → DingTalk / Feishu / PagerDuty / Email.

4.2 Push vs Pull

Pull suits long‑lived services exposing /metrics. It fails for short‑lived jobs that finish before the scrape interval.

Push is recommended for short‑lived or one‑off jobs (K8s Job, CronJob) and for heterogeneous language environments using an OpenTelemetry Collector.

Metric Design

Lifecycle Metrics

batch_task_run_total{job_name="settlement_job",status="started"}
batch_task_run_total{job_name="settlement_job",status="succeeded"}
batch_task_run_total{job_name="settlement_job",status="failed"}
batch_task_running{job_name="settlement_job"}
batch_task_start_timestamp_seconds{job_name="settlement_job"}
batch_task_end_timestamp_seconds{job_name="settlement_job"}

Progress Metrics

batch_task_total_records{job_name="settlement_job",run_id="202605300100"}
batch_task_processed_records_total{job_name="settlement_job",run_id="202605300100"}
batch_task_failed_records_total{job_name="settlement_job",run_id="202605300100"}
batch_task_skipped_records_total{job_name="settlement_job",run_id="202605300100"}
batch_task_checkpoint_offset{job_name="settlement_job",shard_id="3"}

Derived success rate:

100 * sum(batch_task_processed_records_total) / sum(batch_task_total_records)

.

Throughput Metrics

rate(batch_task_processed_records_total[1m])
rate(batch_task_batch_total[1m])
batch_task_output_bytes_total
rate(batch_task_output_bytes_total[1m])

Monitor records/s, batches/s, and MB/s because record size may spike.

Latency Metrics

batch_task_record_duration_seconds_bucket
batch_task_batch_duration_seconds_bucket
batch_task_e2e_duration_seconds_bucket

Single‑record latency helps locate local anomalies; batch latency points to write or remote‑call slowdowns; end‑to‑end latency serves SLA reporting.

Quality Metrics

batch_task_retry_total{job_name="settlement_job"}
batch_task_dead_letter_total{job_name="settlement_job"}
batch_task_duplicate_total{job_name="settlement_job"}
batch_task_dirty_record_total{job_name="settlement_job"}
batch_task_idempotent_conflict_total{job_name="settlement_job"}

Critical for finance, payment, marketing‑coupon, and billing scenarios.

Resource Metrics

batch_task_worker_inflight
batch_task_queue_depth
batch_task_db_connections_in_use
batch_task_external_request_inflight
process_resident_memory_bytes
go_goroutines
jvm_threads_live_threads
jvm_gc_pause_seconds_bucket

Observability must link task‑level metrics with host‑process resources; otherwise you only know "slow" without the cause.

Logging Strategy

Control‑Plane Logs

Record lifecycle events (task start, end, shard assignment, checkpoint commit, cancellation) in structured JSON.

{
  "level":"INFO",
  "event":"task_started",
  "job_name":"settlement_job",
  "run_id":"202605300100",
  "schedule_time":"2026-05-30T01:00:00+08:00",
  "data_window":"2026-05-29",
  "worker_count":32
}

Data‑Plane Exception Logs

Log only important failure samples (dirty data, idempotent conflict, downstream timeout, retry exhaustion) with sampling to avoid log explosion.

Audit & Compensation Logs

Include fields such as job_name, run_id, shard_id, batch_id, record_key, error_type, error_code, retry_count, checkpoint to support replay and manual investigation.

Tracing Integration

Use spans for each logical step (task.run → shard.process → source.fetch → batch.transform → sink.write → checkpoint.commit). Attach job_name, run_id, shard_id, batch_id, attempt, record_count to every span.

task.run
 -> shard.process
   -> source.fetch
   -> batch.transform
   -> sink.write
   -> checkpoint.commit

Link metrics and logs to traces via run_id (metric label) and trace_id / span_id (log fields). Grafana data‑links can jump from an alert to the relevant trace and log.

Alerting Design

Four‑level alert model (P0–P3) with concrete PromQL rules:

P0 – task not started or exceeds business window.

P1 – severe degradation (high error rate, dead‑letter surge, multiple shard failures).

P2 – risk warning (throughput drop, retry storm).

P3 – informational (non‑core task delay, compensation start).

Example alerts:

# Task not started within 10 min
(time() - batch_task_start_timestamp_seconds{job_name="settlement_job"} > 600) and batch_task_running{job_name="settlement_job"} == 0

# Progress stalled
increase(batch_task_processed_records_total{job_name="settlement_job"}[10m]) == 0 and batch_task_running{job_name="settlement_job"} == 1

# Throughput drop
sum(rate(batch_task_processed_records_total{job_name="settlement_job"}[5m])) < 0.5 * avg_over_time(sum(rate(batch_task_processed_records_total{job_name="settlement_job"}[5m]))[30m:5m])

# Error rate >5%
sum(rate(batch_task_failed_records_total{job_name="settlement_job"}[5m])) / clamp_min(sum(rate(batch_task_processed_records_total{job_name="settlement_job"}[5m])),1) > 0.05

# Retry storm >200 per 5 min
sum(rate(batch_task_retry_total{job_name="settlement_job"}[5m])) > 200

# Exceeds business window (e.g., must finish before 03:00)
batch_task_running{job_name="settlement_job"} == 1 and (time() - batch_task_start_timestamp_seconds{job_name="settlement_job"} > 7200)

Group alerts by job_name, run_id, and severity, suppress lower‑severity alerts when a higher‑severity one fires, and apply for‑duration debounce (e.g., 3‑10 min for throughput, 2‑5 min for error rate, 5 min for stall).

High‑Throughput Engineering Practices

Count everything with Counters; sample latency with native histograms or low‑rate sampling.

Aggregate locally and push every 15‑30 s; flush on task completion.

Keep short‑lived labels ( run_id) only while the task runs; delete them from Pushgateway after exit.

Go SDK Example (Simplified)

package batchobs

import (
    "context"
    "log/slog"
    "math/rand"
    "os"
    "sync/atomic"
    "time"
    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/push"
)

type TaskContext struct {
    JobName      string
    RunID        string
    ScheduleTime string
    DataWindow   string
    TenantID     string
    ShardID      string
    Attempt      int
    Instance     string
}

type Config struct {
    PushgatewayURL string
    PushInterval   time.Duration
    SampleRatio    float64
}

// Observer implements lifecycle, progress, retry, and resource metrics.
// It registers all metrics with a Prometheus registry and pushes them via Pushgateway.
// The implementation follows the design described in the article.

The SDK records start/end timestamps, total/processed/failed counters, queue depth gauge, record and batch histograms, and provides periodic flush and cleanup methods.

Java SDK Sketch (Spring Boot + Micrometer + OpenTelemetry)

package com.example.batch.obs;

import io.micrometer.core.instrument.*;
import io.opentelemetry.api.trace.*;
import java.time.Instant;
import java.util.concurrent.atomic.*;

public class BatchTaskObserver {
    private final MeterRegistry registry;
    private final Tracer tracer;
    private final String jobName;
    private final String runId;
    private final AtomicLong totalRecords = new AtomicLong();
    private final AtomicLong processedRecords = new AtomicLong();
    private final AtomicInteger queueDepth = new AtomicInteger();
    private final Counter processedCounter;
    private final Counter failedCounter;
    private final Counter retryCounter;
    private final Timer recordTimer;
    private final Timer batchTimer;

    public BatchTaskObserver(MeterRegistry registry, Tracer tracer, String jobName, String runId) {
        this.registry = registry;
        this.tracer = tracer;
        this.jobName = jobName;
        this.runId = runId;
        Gauge.builder("batch_task_total_records", totalRecords, AtomicLong::get)
            .tag("job_name", jobName).tag("run_id", runId).register(registry);
        Gauge.builder("batch_task_queue_depth", queueDepth, AtomicInteger::get)
            .tag("job_name", jobName).tag("run_id", runId).register(registry);
        this.processedCounter = Counter.builder("batch_task_processed_records_total").tag("job_name", jobName).tag("run_id", runId).register(registry);
        this.failedCounter = Counter.builder("batch_task_failed_records_total").tag("job_name", jobName).tag("run_id", runId).register(registry);
        this.retryCounter = Counter.builder("batch_task_retry_total").tag("job_name", jobName).tag("run_id", runId).register(registry);
        this.recordTimer = Timer.builder("batch_task_record_duration_seconds").publishPercentileHistogram().tag("job_name", jobName).tag("run_id", runId).register(registry);
        this.batchTimer = Timer.builder("batch_task_batch_duration_seconds").publishPercentileHistogram().tag("job_name", jobName).tag("run_id", runId).register(registry);
    }
    // start, setQueueDepth, recordProcessed, recordFailed, recordRetry, traceBatch methods omitted for brevity.
}

The Java observer mirrors the Go design, exposing counters, gauges and timers with the same label set.

Dashboard Recommendations

Design Grafana panels in the order of troubleshooting:

Overall view: running flag, progress percent, current throughput, error rate.

Shard & batch view: slowest shard_id, highest failure shard, checkpoint stagnation.

Resource view: thread‑pool usage, DB connection pool, GC pause, external latency.

Log & trace links: jump to Loki with run_id, jump to Tempo/Jaeger with the same identifier.

Sample queries:

100 * sum(batch_task_processed_records_total{job_name=~"$job"}) / clamp_min(sum(batch_task_total_records{job_name=~"$job"}),1)

sum by(shard_id)(rate(batch_task_processed_records_total{job_name=~"$job"}[$__rate_interval]))

histogram_quantile(0.99, sum by(le,shard_id)(rate(batch_task_batch_duration_seconds_bucket{job_name=~"$job"}[$__rate_interval])))

Common Pitfalls

Pushgateway metric residue – clean up with defer Cleanup() and periodic deletion.

High‑cardinality labels (record_id, user_id, order_id, trace_id) cause storage explosion; keep only low‑cardinality identifiers like job_name, run_id, shard_id, status, error_type, tenant_id.

Ignoring skipped, dirty or dead‑letter counts makes a task appear successful while data is incomplete.

Only measuring total latency hides which stage (fetch, transform, write) is the bottleneck.

Alert thresholds not aligned with business SLA (e.g., task finishes after the required reporting window) cause missed business impact.

Evolution Path from 0 to 100 M Records

Stage 1 – single‑task monitoring (status, total time, basic logs).

Stage 2 – task‑level metric suite (progress, throughput, error rate, shard view).

Stage 3 – distributed tracing across services, linking metrics and logs via run_id.

Stage 4 – platform‑wide governance: unified SDK, alert templates, registration center, compensation platform, dynamic thresholding.

Best‑Practice Checklist

Define a unified identifier set: job_name / run_id / shard_id / attempt / checkpoint.

Embed the state machine in the platform or SDK.

Ensure checkpoints and compensation are built into the execution flow.

Expose six metric families (lifecycle, progress, throughput, latency, quality, resource) with proper Counter/Gauge/Histogram usage.

Strictly avoid high‑cardinality labels.

Emit full control‑plane logs; sample data‑plane error logs.

Make logs searchable by run_id.

Structure spans into task → shard → batch levels and attach context tags.

Link metrics, logs and traces for one‑click navigation.

Implement six alert categories (not started, stalled, error‑rate, throughput drop, timeout, retry storm) with grouping, suppression and debounce.

Aggregate metrics locally before pushing; clean up short‑lived metrics after task exit.

Route failed records to dead‑letter or compensation pipelines.

Conclusion

Batch jobs often fail silently by drifting from expected performance. A mature observability system combines a unified state machine, layered metrics, structured logs, distributed tracing and tiered alerts to turn a black‑box script into an operable, scalable production service. When an alert fires, engineers should instantly know which job, run and shard failed, why it stalled or degraded, which resource is the bottleneck, whether the business window is breached, and what remediation (retry, compensation, scaling, rollback) is required.