Spring Cloud Microservices in Practice – Revised Part 7: Using SkyWalking for Distributed Tracing

Goal

This article addresses the remaining challenge of problem diagnosis in a Spring Cloud microservice system by introducing SkyWalking for distributed tracing.

Why Distributed Tracing?

Identify which service is slow when a request passes through multiple services.

Locate the exact component (database, code, etc.) causing errors.

Visualize complex service dependencies.

Environment Preparation

1. Start SkyWalking services (OAP, Elasticsearch, UI) with Docker Compose

version: '3.8'
services:
  elasticsearch:
    image: elasticsearch:7.17.0
    container_name: elasticsearch
    environment:
      - discovery.type=single-node
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ports:
      - "9200:9200"
    networks:
      - teaching-network

  oap:
    image: apache/skywalking-oap-server:9.7.0
    container_name: oap
    depends_on:
      - elasticsearch
    environment:
      - SW_STORAGE=elasticsearch
      - SW_STORAGE_ES_CLUSTER_NODES=elasticsearch:9200
    ports:
      - "11800:11800"
      - "12800:12800"
    networks:
      - teaching-network

  ui:
    image: apache/skywalking-ui:9.7.0
    container_name: ui
    depends_on:
      - oap
    environment:
      - SW_OAP_ADDRESS=http://oap:12800
    ports:
      - "8088:8080"
    networks:
      - teaching-network

networks:
  teaching-network:
    driver: bridge

2. Download the matching SkyWalking Java Agent

# Download Agent (match OAP version)
wget https://archive.apache.org/dist/skywalking/9.7.0/apache-skywalking-java-agent-9.7.0.tgz
# Extract
tar -zxvf apache-skywalking-java-agent-9.7.0.tgz
# Directory layout after extraction
# skywalking-agent/
# ├── skywalking-agent.jar
# ├── config/
# └── plugins/

Service Integration

1. Add Java agent parameters to each service startup command

# user-service
java -javaagent:/path/to/skywalking-agent/skywalking-agent.jar \
     -Dskywalking.agent.service_name=user-service \
     -Dskywalking.collector.backend_service=localhost:11800 \
     -jar user-service.jar

# order-service
java -javaagent:/path/to/skywalking-agent/skywalking-agent.jar \
     -Dskywalking.agent.service_name=order-service \
     -Dskywalking.collector.backend_service=localhost:11800 \
     -jar order-service.jar

# gateway
java -javaagent:/path/to/skywalking-agent/skywalking-agent.jar \
     -Dskywalking.agent.service_name=gateway \
     -Dskywalking.collector.backend_service=localhost:11800 \
     -jar gateway.jar

2. Maven plugin method (recommended for development)

<build>
  <plugins>
    <plugin>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-maven-plugin</artifactId>
      <configuration>
        <jvmArguments>
          -javaagent:${project.basedir}/../skywalking-agent/skywalking-agent.jar \
          -Dskywalking.agent.service_name=${project.artifactId} \
          -Dskywalking.collector.backend_service=localhost:11800
        </jvmArguments>
      </configuration>
    </plugin>
  </plugins>
</build>

3. IDEA configuration (VM options)

-javaagent:/path/to/skywalking-agent/skywalking-agent.jar
-Dskywalking.agent.service_name=user-service
-Dskywalking.collector.backend_service=localhost:11800

Agent Configuration Details

Key entries in skywalking-agent/config/agent.config:

agent.service_name=${SW_AGENT_NAME:your-service-name}

collector.backend_service=${SW_AGENT_COLLECTOR_BACKEND_SERVICES:localhost:11800}

Sampling rate (recommended 10% in production): agent.sample_n_per_3_secs=10 Ignore static resources: agent.ignore_suffix=.jpg,.jpeg,.png,.css,.js Log level: logging.level=${SW_LOGGING_LEVEL:INFO} Record SQL parameters: plugin.mysql.trace_sql_parameters=true Collect HTTP request parameters in Spring MVC:

plugin.springmvc.collect_http_params=true

Verification of Tracing

1. Start all services

# Start Nacos, Sentinel, SkyWalking
docker-compose up -d

# Start each Spring Boot service with the agent
cd user-service && mvn spring-boot:run
cd order-service && mvn spring-boot:run
cd gateway && mvn spring-boot:run

2. Generate test requests

# Repeatedly call the order API to produce trace data
for i in {1..10}; do
  curl http://localhost:8080/api/order/1
  sleep 1
done

3. View tracing results in SkyWalking UI

Open http://localhost:8088 and explore:

Topology : shows service dependency graph (e.g., gateway → order-service → user-service).

Trace List : lists all request traces with Trace ID, endpoint, latency, and status.

Span Details : click a trace to see a timeline, pinpoint slow spans (e.g., a slow DB query in user-service).

Spring Boot Integration (Zero‑Intrusion)

The SkyWalking agent automatically supports the following plugins without code changes:

Spring MVC – controller method tracing

Spring Cloud Gateway – gateway routing tracing

OpenFeign – Feign client call tracing

RestTemplate – HTTP call tracing

MySQL Driver – SQL execution tracing

Redis Client – Redis operation tracing

ThreadPool – thread‑pool task tracing

Alarm Configuration

1. Built‑in alarm rules (oap/config/alarm-settings.yml)

rules:
  # Service response time alarm
  service_resp_time_rule:
    metrics-name: service_resp_time
    op: ">"
    threshold: 1000
    period: 3
    count: 3
    message: "Service {name} response time exceeds 1000ms"

  # Service SLA alarm
  service_sla_rule:
    metrics-name: service_sla
    op: "<"
    threshold: 95
    period: 3
    count: 2
    message: "Service {name} success rate below 95%"

  # Database slow query alarm
  database_resp_time_rule:
    metrics-name: database_resp_time
    op: ">"
    threshold: 500
    period: 2
    count: 3
    message: "Database query exceeds 500ms"

2. DingTalk webhook example

webhooks:
  - https://oapi.dingtalk.com/robot/send?access_token=xxx

Common Issues and Troubleshooting

Issue 1: Agent not effective

Verify the -javaagent path is correct.

Check that the OAP address is reachable (e.g., telnet localhost 11800).

Inspect the agent log file ( logs/skywalking-api.log).

Issue 2: Incomplete trace

Cause: missing plugin for a service/component.

Solution: ensure the required plugin JAR exists under skywalking-agent/plugins/ (e.g., apm-spring-cloud-gateway-3.x-plugin, apm-feign-default-http-9.x-plugin, apm-mysql-8.x-plugin).

Issue 3: UI shows no data

Confirm both OAP and UI containers are running.

Verify the agent is successfully reporting (check OAP logs).

Ensure system clocks of agent and OAP are synchronized.

Code Retrieval

No source code changes are required for this part; only the Java agent parameters need to be added at startup.

Supported Services List

user-service – -Dskywalking.agent.service_name=user-service order-service – -Dskywalking.agent.service_name=order-service gateway –

-Dskywalking.agent.service_name=gateway

Full Docker‑Compose Configuration (for reference)

version: '3.8'
services:
  nacos:
    image: nacos/nacos-server:v2.3.2
    container_name: nacos-teaching
    ports:
      - "8848:8848"
      - "9848:9848"

  sentinel:
    image: bladex/sentinel-dashboard:1.8.6
    container_name: sentinel-teaching
    ports:
      - "8858:8858"

  elasticsearch:
    image: elasticsearch:7.17.0
    container_name: elasticsearch
    environment:
      - discovery.type=single-node
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ports:
      - "9200:9200"

  oap:
    image: apache/skywalking-oap-server:9.7.0
    container_name: oap
    depends_on:
      - elasticsearch
    environment:
      - SW_STORAGE=elasticsearch
      - SW_STORAGE_ES_CLUSTER_NODES=elasticsearch:9200
    ports:
      - "11800:11800"
      - "12800:12800"

  ui:
    image: apache/skywalking-ui:9.7.0
    container_name: ui
    depends_on:
      - oap
    environment:
      - SW_OAP_ADDRESS=http://oap:12800
    ports:
      - "8088:8080"

networks:
  default:
    driver: bridge

Next Episode Preview

Spring Cloud Microservices – Revised Part 8: Seata Distributed Transactions (covering distributed transaction problems, AT mode, TCC mode, and hands‑on exercises).