Production-Ready Observability: Spring Boot 3.x, Actuator, Prometheus & Grafana

1. Concept Upgrade: Monitoring vs Observability

Monitoring tells you if the system is alive; observability tells you why it behaves abnormally. Production observability rests on three pillars that must interconnect:

Metrics : Aggregated time-series data (QPS, latency, CPU, memory, GC). Low overhead, ideal for real-time alerting to quickly isolate which service or endpoint has issues.

Traces : Full request call chains across distributed systems. They pinpoint where a request stalls — gateway, order service, or Redis.

Logs : Discrete event records with business context and stack traces. They uncover root causes but must be structured and share the same TraceId with metrics and traces.

When these three operate in silos, troubleshooting forces context-switching across tools. The correct flow: metrics alert triggers, use TraceId to inspect the call chain in Jaeger/Zipkin, identify a slow downstream service, then filter logs in Kibana/Loki with the same TraceId. Unified data sources and identifiers close the loop.

2. Actuator Endpoint Configuration & Security Hardening

2.1 Core Configuration

management:
  endpoints:
    web:
      exposure:
        include: health,info,metrics,prometheus  # expose only needed
      base-path: /actuator
    endpoint:
      health:
        show-details: when-authorized  # never 'always' in production
      probes:
        enabled: true  # required for K8s native probes
  server:
    port: 8081  # separate monitoring port from business traffic to prevent DDoS or saturation

Kubernetes livenessProbe and readinessProbe automatically map to /actuator/health/liveness and /actuator/health/readiness.

2.2 Security Hardening

Authentication : Protect /actuator/** with Basic Auth or OAuth2 via Spring Security; unauthenticated requests return 401.

Network Isolation : Use security groups or K8s NetworkPolicy to whitelist only Prometheus server and bastion host IPs.

Path Obfuscation : Change base-path to a random string like /sys-metrics-x9k2 to defeat automated scanners.

2.3 Custom Health Indicators

Default checks only verify DB/Redis connectivity. Business-critical states (order queue depth, third-party channel switches) require custom HealthIndicator implementations:

@Component
public class OrderQueueHealthIndicator implements HealthIndicator {
  @Override
  public Health health() {
    long pendingCount = orderService.getPendingQueueSize();
    if (pendingCount > 10000) {
      return Health.down().withDetail("queue_depth", pendingCount).build();
    }
    return Health.up().withDetail("queue_depth", pendingCount).build();
  }
}

When status turns DOWN, K8s readiness probe removes the pod from service traffic. Combined with auto-restart or scaling, the system self-heals.

3. Metrics Collection: Micrometer Instrumentation & Prometheus Scraping

3.1 Dependency

<dependency>
  <groupId>io.micrometer</groupId>
  <artifactId>micrometer-registry-prometheus</artifactId>
</dependency>

After restart, /actuator/prometheus exposes Prometheus-formatted plain text.

3.2 Auto-Collection & Cardinality Explosion Trap

Framework auto-collects JVM memory, GC pauses, HTTP request latency. The production pitfall is cardinality explosion . Prometheus storage is highly sensitive to tag cardinality. Never put unbounded fields like userId, orderId, clientIp into tags — metric count grows exponentially, causing OOM. HTTP uri tags must be normalized; /api/v1/orders/123 and /api/v1/orders/456 are distinct series.

Spring Boot 3 deprecates WebMvcTagsProvider in favor of the Observation API. Implement HttpServerObservationConvention or register an ObservationRegistryCustomizer to regex-replace numeric path segments with placeholders, stopping cardinality growth at the source.

3.3 Business Instrumentation Example

@Service
public class PaymentService {
  private final Timer paymentTimer;
  private final Counter paymentFailedCounter;

  public PaymentService(MeterRegistry registry) {
    paymentTimer = Timer.builder("payment.duration.seconds")
      .description("Payment core path latency")
      .publishPercentiles(0.5, 0.95, 0.99)  // enable only needed percentiles
      .register(registry);

    paymentFailedCounter = Counter.builder("payment.failed.count")
      .tag("reason", "timeout")
      .register(registry);
  }

  public void execute(PaymentRequest req) {
    paymentTimer.record(() -> {
      try { doPay(req); }
      catch (TimeoutException e) { paymentFailedCounter.increment(); throw e; }
    });
  }
}

Excessive publishPercentiles increases client-side computation; retain only 95th and 99th percentiles in production.

3.4 Thread Pool Metrics Exposure

Custom ThreadPoolTaskExecutor beans are not auto-registered. Bind manually via MeterBinder:

@Bean
public MeterBinder threadPoolMetrics(ThreadPoolTaskExecutor executor) {
  return registry -> {
    Gauge.builder("thread.pool.active.count", executor,
      e -> e.getThreadPoolExecutor().getActiveCount())
      .tag("name", "async-executor").register(registry);
    Gauge.builder("thread.pool.queue.size", executor,
      e -> e.getThreadPoolExecutor().getQueue().size())
      .tag("name", "async-executor").register(registry);
  };
}

Thread pool saturation is often more fatal than 100% CPU; exposing these gauges reveals queue buildup trends early.

3.5 Prometheus Scrape Config Correction

In Kubernetes, dynamic discovery is standard. The original relabel_configs had syntax issues; here is a production-ready snippet:

scrape_configs:
- job_name: 'spring-boot-apps'
  metrics_path: '/actuator/prometheus'
  scrape_interval: 15s
  scrape_timeout: 10s
  kubernetes_sd_configs:
  - role: pod
  relabel_configs:
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
    action: keep
    regex: true
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
    action: replace
    target_label: __metrics_path__
    regex: (.+)
  - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
    action: replace
    regex: ([^:]+)(?::\d+)?;(\d+)
    replacement: $1:$2
    target_label: __address__

Controlling scrape targets via pod annotations is far more flexible than hard-coded selectors, eliminating repeated Prometheus config changes for both ops and devs.

4. Visualization: Grafana Dashboards That Avoid Pitfalls

Grafana is the de facto standard, but blindly importing community templates clutters dashboards with unused panels. Production dashboards should be minimal and follow the RED methodology (Rate, Errors, Duration).

QPS & Traffic Baseline :

sum(rate(http_server_requests_seconds_count{service="$svc"}[5m])) by (uri)

. A 5-minute sliding window smooths momentary spikes. Sudden cliff or doubling warrants checking downstream gateway or cache layer first.

Latency Percentiles (P95/P99) : HTTP latency is a Histogram in Micrometer. Use

histogram_quantile(0.99, sum(rate(http_server_requests_seconds_bucket{service="$svc"}[5m])) by (le, uri))

. Never rely solely on average latency — it gets diluted by fast requests; the 1% long tail hurts user experience.

Error Rate & Resource Saturation : Error rate = 5xx rate / total request rate. Memory and CPU use jvm_memory_used_bytes and system collectors. Memory leak alerts must combine current usage with jvm_gc_pause_seconds_count and GC duration. If Full GC frequency shifts from a few per day to every few minutes, intervene even before OOM.

Avoid static alert thresholds. Use historical percentiles for baseline alerts, e.g.,

P99 latency exceeds 2x the same time window's P99 over the past 7 days, sustained for 3 minutes

. Route alerts by severity: P0 triggers phone+SMS, P1 goes to WeCom/DingTalk bots, P2 logs to ticketing. Weekly prune false positives to combat alert fatigue.

5. Distributed Tracing: Micrometer Tracing & OpenTelemetry

Spring Cloud Sleuth is retired in Spring Boot 3; the official replacement is Micrometer Tracing backed by OpenTelemetry.

5.1 Dependencies & Protocol

<dependency>
  <groupId>io.micrometer</groupId>
  <artifactId>micrometer-tracing-bridge-otel</artifactId>
</dependency>
<dependency>
  <groupId>io.opentelemetry</groupId>
  <artifactId>opentelemetry-exporter-otlp</artifactId>
</dependency>

OTLP is the most widely supported export protocol; Jaeger, Tempo, SkyWalking all support it natively. Abandon legacy Zipkin HTTP v2.

5.2 Context Propagation & Async Breaks

Distributed tracing relies on the traceparent header. Spring MVC, WebClient, RestTemplate auto-intercept and propagate for most HTTP calls.

The main breakage points are async threads and message queues . @Async thread pools lose trace context unless propagated. Micrometer Tracing offers @Observed annotation and ObservationFilter, or wrap Runnable/Callable with TracingContextPropagator. Kafka/RabbitMQ listeners must configure header propagation; otherwise the trace ends at the consumer.

5.3 Sampling Strategy & Jaeger Deployment

Full sampling in production is suicidal — at least 30% performance overhead and unsustainable storage costs. Production typically uses head-based sampling at 1%-5%, sufficient to capture most abnormal paths:

management:
  tracing:
    sampling:
      probability: 0.05
  zipkin:
    tracing:
      endpoint: "http://otel-collector:4318/v1/traces"  # OTLP endpoint

For core flows (payment, order) requiring full tracing, implement a custom Sampler that dynamically adjusts by URL or header instead of changing global config. Deploy Jaeger behind an OTel Collector for batching, tail sampling, and format conversion; never let business pods hit Jaeger directly.

6. Log Governance: Structured Output & MDC Propagation

Traditional pattern %d [%t] %-5p %c - %m%n becomes a disaster in ELK/Loki — unstructured logs prevent efficient search and correlation.

6.1 Logback JSON Transformation

Add logstash-logback-encoder to emit single-line JSON. Micrometer Tracing automatically places traceId and spanId into MDC; just declare MDC inclusion in the encoder:

<encoder class="net.logstash.logback.encoder.LogstashEncoder">
  <customFields>{"service":"order-service","env":"${ENV_NAME}"}</customFields>
  <includeMdc>true</includeMdc>
</encoder>

With includeMdc=true, each log line gains an mdc field containing trace IDs. Collectors parse this directly without complex Grok regex, boosting efficiency.

6.2 Collection Pipeline & Cost Reduction

Standard pipeline: App → Filebeat/FluentBit → Kafka → Logstash/Vector → ES → Kibana. Enable json.keys_under_root: true in Filebeat to prevent JSON nesting under json.log which breaks field expansion. ES storage is the major cost; apply ILM policies: hot nodes retain 3-7 days for recent issues, warm nodes compress and archive 30 days, then delete or move to cold storage. Never push full DEBUG logs to production; default to INFO, and use Actuator's /loggers endpoint for dynamic level changes when debugging.

6.3 Troubleshooting Workflow

Incident response must be deterministic: Alert pushes TraceId → click into Jaeger to find red spans with high latency → identify slow SQL or downstream timeout → copy TraceId into Kibana to filter all logs for that request → correlate input parameters and stack traces to locate code or config issue. Codify this into a runbook so newcomers can triage within 5 minutes. For known retry-induced WARN logs, deduplicate or downgrade at collection layer to avoid alert storms.

7. Production Launch Checklist & SRE Practices

Observability isn't a one-time deployment; it must evolve with the business. Pre-launch verification eliminates 80% of production disputes.

Actuator Layer : Monitoring port isolated from business. /health detail requires authentication. K8s readiness probe must verify dependent middleware (e.g., Redis) to avoid false healthy pods.

Metrics Layer : All custom metrics follow snake_case. Audit for high-cardinality tags. Core endpoint timer coverage must reach 100% — no blind spots.

Prometheus Layer : Set retention (minimum 15 days) per disk capacity. Single node insufficient? Add Thanos or VictoriaMetrics for long-term storage and federation. Scrape timeout not too short; 10 seconds accommodates GC pauses.

Tracing Layer : Start sampling at 5%, observe RT impact, then fine-tune. Verify context propagation across all async threads, scheduled tasks, MQ consumers; fix broken chains with @Observed or manual wrapping.

Logging Layer : Confirm JSON parses correctly without nested explosions. ERROR logs must trigger real-time alerts. traceId field from MDC must appear in both log templates and alert templates for one-click jump.

SRE Daily Recommendations : Don't blindly pile metrics. Define SLO first (e.g., 99.95% availability), derive SLIs (success rate, latency percentiles), then map to concrete Prometheus queries. Follow Google's Four Golden Signals: latency, traffic, errors, saturation. Once these four dimensions are solid, dashboards stand firm. Alerts must be denoised — weekly review false positives, replace static thresholds with dynamic baselines or percentile comparisons. Run regular chaos drills: inject dependency timeouts or pod evictions, verify alerts fire, dashboards react, and on-call engineers can mitigate within MTTR using runbooks.

The system launch is only the starting point. Production always presents unexpected edge cases. Maintain data sensitivity, feed every troubleshooting lesson back into monitoring rules, and the stack becomes smoother with use.