Seeing the Full Request Journey: Completing Spring Boot Trace Integration

Why Metrics and Logs Are Not Sufficient

Metrics (e.g., HTTP QPS, P99 latency) show aggregated trends, and logs show individual error lines, but neither can answer the question “which part of this request took how long?”. Traces fill that gap by recording the complete request journey across services.

Core Concepts: Trace, Span, Parent/Child

A request entering a gateway may travel through Service A → Service B → Database. Tracing records this journey as a tree:

TraceId: abc123
│
├─ Span: GET /api/chain (app-monitor, 120ms)
│   ├─ Span: order.create (app-monitor, 80ms)   ← business Span
│   └─ Span: GET /api/echo (monitor-downstream, 40ms)

Each Span typically carries service name, operation name, duration, status, and a few low‑cardinality tags (e.g., order.type=online). High‑cardinality identifiers such as orderId or userId should be avoided.

How Traces Are Generated (Java Stack)

┌─────────────────────────────────────┐
│  Spring Boot process                │
│ ① Java Agent attaches to JVM → auto‑intercepts HTTP inbound/outbound, thread‑context propagation
│ ② Optional business code → manual Local Span (e.g., order.create)
│ ③ Cross‑service HTTP call → Agent injects Trace context via headers → downstream creates child Span (TraceId unchanged)
└───────────────────────┬─────────────┘
                        ▼
              ┌─────────────────────┐
              │  OAP (Observability │
              │  Analysis Platform)│
              │  → aggregates Segments, assembles Trace, writes to storage (BanyanDB, etc.) │
              └─────────────────────┘
                        ▼
              SkyWalking UI / Grafana query UI

Key points:

Automatic instrumentation is the default; the Java Agent enhances Spring MVC, RestTemplate, WebClient, etc., with zero code changes.

Context propagation passes the TraceId via HTTP headers so that downstream spans belong to the same Trace.

Sampling is usually partial in production to limit overhead; demos can use 100 % sampling.

Linking Traces with Metrics and Logs

Logs → Trace : Include [TID:abc123] in log lines; clicking the link jumps to the corresponding Trace.

Trace → Logs : Query Loki by TraceId and time range to retrieve related logs.

Metrics → Trace : When a metric (e.g., P99) spikes, use the same time window to list relevant Traces.

Dashboard time‑axis alignment with a Trace list is sufficient for demo scenarios.

Why Choose SkyWalking

Comparison of three Java‑centric tracing solutions:

SkyWalking : Mature Java ecosystem, non‑intrusive agent, built‑in UI and topology view; ideal for Java/Spring Boot services.

Grafana Tempo + OpenTelemetry : Native Grafana integration and Prometheus exemplars; better for multi‑language stacks already using OTel.

Jaeger : Classic CNCF project with strong multi‑language support and Kubernetes‑native deployment.

For an existing Prometheus + Grafana + Loki stack, SkyWalking was selected because its agent supports Spring Boot 3.x / Java 21, it provides an out‑of‑the‑box OAP + UI, and its Logback toolkit injects TraceId automatically.

Overall Architecture: Adding a Layer

┌──────────────────────────────────────────────────────────────────┐
│  Application layer (independent docker‑compose)                 │
│  app‑monitor (:8882) ──HTTP──▶ monitor‑downstream (:8883)      │
│       │  + Java Agent          │  + Java Agent                │
└───────┼───────────────────────┼─────────────────────────────┘
        │ gRPC reporting        │
        ▼                       ▼
┌──────────────────────────────────────────────────────────────────┐
│  monitoring/docker‑compose.yml (shared monitoring stack)          │
│  Prometheus ── Grafana ── Loki/Promtail (existing)               │
│        │               └── SkyWalking data‑source plugin          │
│  SkyWalking OAP ── BanyanDB (Trace storage)                     │
│  SkyWalking UI (:8080)                                          │
└──────────────────────────────────────────────────────────────────┘

Design principles remain the same as the previous articles: a single Compose file provides the monitoring infrastructure for multiple applications, and each app only needs to mount the agent and set SW_AGENT_NAME.

Application Configuration

Set the OAP address (e.g., skywalking‑oap:11800).

Define the service name (e.g., app‑monitor); multiple instances can share the same name.

Configure sampling rate (100 % for demos).

No code changes to Controllers or Filters are required—the Agent handles HTTP entry, outbound client calls, and thread‑context propagation automatically.

When to Add Manual Business Spans

Automatic spans show that /api/orders took 200 ms, but they cannot tell how much time each business step consumed. Adding manual spans such as order.create and payment.process creates additional layers in the Trace tree, making it clear whether the delay is in DB access or business logic.

Span names should reflect business semantics, not class or method names.

Only low‑cardinality tags (e.g., order.type, payment.channel) should be attached.

The domain layer should not depend directly on SkyWalking APIs; use adapters to keep DDD friendliness.

In the SkyWalking UI, the Trace for POST /api/orders expands to three layers: the HTTP entry, order.create, and payment.process, with a timeline that visualises each segment’s duration.

Log Integration

After adding the SkyWalking Logback toolkit, every log line automatically includes [TID:…]. Loki collects these logs, and Grafana’s derived‑field configuration adds a “View Trace” link that jumps from an ERROR log to the corresponding Trace in two clicks.

Cross‑Service Tracing

A downstream service monitor‑downstream is added to demonstrate cross‑process collaboration. The Trace tree then contains spans from both services, and SkyWalking’s Topology view draws the dependency app‑monitor → monitor‑downstream.

Opening the Topology tab in SkyWalking UI displays the call chain User → app‑monitor → monitor‑downstream, allowing quick location of slow or failing endpoints.

Where to View Traces

SkyWalking UI – http://localhost:8080: topology, Trace details, service list (preferred).

Grafana Explore – SkyWalking data source: view Traces alongside Metrics and Logs in the same console.

Grafana Dashboard – “App Monitor Tracing” panel: three pillars displayed side‑by‑side with synchronized timelines.

Note: The GraphQL endpoint http://127.0.0.1:12800/graphql only accepts POST; use the UI or Grafana for Trace inspection.

Full Troubleshooting Story

Assume a load test shows the P99 of /api/chain rising. The recommended workflow is:

Metrics – In Grafana, locate the abnormal P99 window (e.g., 16:05‑16:10).

Traces – In SkyWalking UI, filter by service app‑monitor, sort by latency, and open the slowest Trace.

Span Analysis – Identify that 80 % of the time is spent in the downstream service’s HTTP client span.

Logs – In Loki, search for the TraceId within the same time window to see related INFO/ERROR entries.

Conclusion – The bottleneck is the downstream service, not the entry service’s business logic.

Step‑by‑Step Flow

Ensure the first‑two‑article monitoring stack (Prometheus/Grafana/Loki) is running; Grafana should be reachable. cd monitoring && docker compose up -d (includes SkyWalking); verify OAP health endpoint returns OK.

Build and start app‑monitor and monitor‑downstream; both services report health UP. curl http://localhost:8882/api/chain?message=demo – should return JSON.

Open SkyWalking UI ( :8080) → Trace; you should see a cross‑service Span tree.

POST /api/orders – the Trace will contain the manual order.create Span.

Search Loki for the TraceId; you can click “View Trace” from the log entry.

If any step stalls, refer to the repository file specs/003-distributed-tracing/quickstart.md for troubleshooting.

Production Considerations

Sampling – 100 % for demos; 1 %–10 % in production, adjust by traffic.

Agent overhead – Generally <10 % impact on P99; lower sampling if higher.

Storage retention – Align BanyanDB/OAP TTL with Loki/Prometheus retention policies.

Version compatibility – OAP 10.2+ requires BanyanDB 0.8+ (0.7 is incompatible).

Health checks – OAP needs SW_HEALTH_CHECKER=default or UI/Grafana will mark it unhealthy.

Image mirrors – In China, use accelerated mirrors (e.g., DaoCloud) for Agent/OAP images.

Three‑Pillar Recap

Metrics : Micrometer → Prometheus → Grafana – shows overall trends and alerts.

Logs : stdout → Promtail → Loki → Grafana – provides stack traces and error details.

Traces : Agent → SkyWalking OAP → UI/Grafana – reveals the complete path of a single request.

The same app‑monitor example project and shared monitoring stack now cover “numbers”, “logs”, and “traces”, completing the observability puzzle.

Learning Path

Run the Trace demo and view a cross‑service Trace in SkyWalking UI.

Try Log ↔ Trace navigation (find TraceId in an ERROR log).

Correlate Metrics and Traces on a shared timeline.

Optional: tune sampling, add a third service, configure alerts.

Reference Links

Example repository: https://gitcode.com/qq_37953312/app-monitor