When APM Agent Overheads Spike: In‑Depth Comparison of SkyWalking vs Pinpoint

Establish Baseline

1. Verify JVM, application and agent versions

java -version
ps -ef | grep -E '[j]ava|skywalking|pinpoint'
find /opt -maxdepth 4 -type f \( -name 'skywalking-agent.jar' -o -name 'pinpoint-bootstrap.jar' \) -ls

Confirm compatibility of the APM probe with the JDK, framework and application before upgrading the agent.

2. Record process CPU, memory and thread count

PID="<JavaProcessPID>"
ps -p "$PID" -o pid,etime,%cpu,%mem,rss,nlwp,cmd
pidstat -rud -p "$PID" 1 10

Replace <JavaProcessPID> with the target service PID; the baseline should cover a real‑traffic peak.

3. Check JVM start parameters

jcmd <JavaProcessPID> VM.command_line
jcmd <JavaProcessPID> VM.flags
jcmd <JavaProcessPID> VM.system_properties | grep -E 'skywalking|pinpoint|javaagent'

Verify the effective -javaagent and related system properties.

4. Inspect heap and GC state

jcmd <JavaProcessPID> GC.heap_info
jstat -gcutil <JavaProcessPID> 1000 10

Use GC data to judge whether the probe increases allocation pressure; record request volume simultaneously.

5. Examine local threads and sockets

ps -L -p <JavaProcessPID> -o pid,tid,pcpu,stat,comm | sort -k3 -nr | head
ss -tpn | grep '<JavaProcessPID>' || true

Distinguish between business‑level connection pools, probe reporting and backend retries.

6. Capture short‑term JFR evidence

jcmd <JavaProcessPID> JFR.start name=apm-baseline settings=profile duration=60s filename=/tmp/apm-baseline.jfr
jcmd <JavaProcessPID> JFR.check

JFR requires JDK 11+; store the file according to data‑classification rules.

7. Check Native Memory Tracking status

jcmd <JavaProcessPID> VM.native_memory summary 2>&1 | head -n 80

If NMT is disabled, enable it via JVM flags and assess restart risk.

SkyWalking Configuration

8. Confirm javaagent injection method

ps -ef | grep '[j]ava' | grep -- '-javaagent:.*skywalking-agent.jar'
grep -R --line-number 'skywalking-agent.jar' /etc/systemd /opt 2>/dev/null

Ensure each JVM injects the agent only once.

9. Backup and view agent.config

sudo cp -a /opt/skywalking/agent/config/agent.config /opt/skywalking/agent/config/agent.config.$(date +%Y%m%d-%H%M%S)
sudo grep -Ev '^[[:space:]]*($|#)' /opt/skywalking/agent/config/agent.config

10. Set service name and backend address

agent.namespace=<EnvironmentName>
agent.service_name=<ServiceName>
agent.instance_name=<InstanceName>
collector.backend_service=<OAPAddress>:11800

Stable service and instance names avoid mis‑diagnosing probe overhead; the backend address must be reachable.

11. Control fixed sampling count

# Keep at most 100 entry samples per 3 seconds; -1 means unlimited
agent.sample_n_per_3_secs=100

Sampling reduces collection load but may hide low‑frequency errors; adjust based on entry QPS and error budget.

12. Limit span count per trace segment

agent.span_limit_per_segment=300

Prevents unbounded span growth in loops, batch jobs or fan‑out requests.

13. Exclude static‑resource suffixes

agent.ignore_suffix=.jpg,.jpeg,.png,.gif,.css,.js,.ico,.svg,.woff2

Exclude only truly unnecessary static requests; otherwise observability gaps appear.

14. Review enabled plugins

find /opt/skywalking/agent/plugins -maxdepth 1 -type f -name '*.jar' | sort
find /opt/skywalking/agent/optional-plugins -maxdepth 1 -type f -name '*.jar' | sort

Retain only plugins required by the current tech stack; check release notes for compatibility.

15. Move non‑essential plugins out of the load path

sudo install -d -m 0755 /opt/skywalking/agent/plugins-disabled
sudo mv /opt/skywalking/agent/plugins/<PluginFile>.jar /opt/skywalking/agent/plugins-disabled/

Test on a single instance, verify core RPC, DB and HTTP links remain visible, and roll back if needed.

16. Inspect agent logs for connection and enhancement errors

sudo tail -n 200 /opt/skywalking/agent/logs/skywalking-api.log
sudo grep -Ei 'error|warn|connect|plugin|enhance' /opt/skywalking/agent/logs/skywalking-api.log | tail -n 100

Connection failures, plugin errors and retries are direct evidence of root causes.

17. Validate connectivity to OAP

nc -vz <OAPAddress> 11800
ss -tpn | grep '<OAPAddress>' || true

Successful netcat does not guarantee protocol/auth success; use it only to rule out basic network issues.

18. Reduce agent log verbosity

logging.level=info

Avoid long‑term debug logging in production to limit I/O and prevent masking real errors.

Pinpoint Configuration

19. Verify Pinpoint agent injection method

ps -ef | grep '[j]ava' | grep -- '-javaagent:.*pinpoint-bootstrap.jar'
grep -R --line-number 'pinpoint-bootstrap.jar' /etc/systemd /opt 2>/dev/null

20. Backup and read Pinpoint configuration

sudo cp -a /opt/pinpoint-agent/profiles/release /opt/pinpoint-agent/profiles/release.$(date +%Y%m%d-%H%M%S)
sudo grep -R --line-number -E 'applicationName|agentId|sampling|profile' /opt/pinpoint-agent/profiles /opt/pinpoint-agent/pinpoint-root.config 2>/dev/null

21. Set application and instance identifiers

profiler.applicationName=<ServiceName>
profiler.agentId=<StableInstanceId>

applicationName

influences service aggregation; agentId must obey length limits.

22. Use COUNTING sampling

profiler.sampling.type=COUNTING
profiler.sampling.counting.rate=10

After adjusting the rate, verify error requests are still traceable.

23. Disable unnecessary profile collection

profiler.profile.enable=false

Reduces fine‑grained performance data overhead but loses diagnostic detail.

24. Locate Pinpoint agent logs

find /opt/pinpoint-agent -type f -name '*.log' -printf '%p
'
sudo grep -Ei 'error|warn|connect|collector|exception' /opt/pinpoint-agent/logs 2>/dev/null | tail -n 120

25. Check network sessions to collector

ss -tpn | grep -Ei 'pinpoint|<CollectorAddress>' || true
nc -vz <CollectorAddress> 9991

Do not expose probe ports broadly via firewall.

26. Identify the active agent profile

grep -R --line-number 'profile' /opt/pinpoint-agent/pinpoint-root.config /opt/pinpoint-agent/profiles 2>/dev/null | head -n 80

Wrong profile selection means configuration changes have no effect.

Comparison Experiment

27. Uniform javaagent injection via environment variable

# /etc/systemd/system/<ServiceName>.service.d/apm.conf
[Service]
Environment="JAVA_TOOL_OPTIONS=-javaagent:/opt/<AgentPath>/<AgentFile>.jar"

Confirm no conflict with existing javaagent or heap parameters.

28. Verify systemd injection result

sudo systemctl daemon-reload
sudo systemctl show <ServiceName> -p Environment --no-pager
sudo systemctl cat <ServiceName>

These commands do not restart the service; restart instances individually after confirmation.

29. Record key metrics before and after changes

# Use actual exporter metrics
sum(rate(http_server_requests_seconds_count{service="<ServiceName>"}[5m]))
sum(rate(jvm_gc_pause_seconds_sum{service="<ServiceName>"}[5m]))

Fix traffic window, JDK, app version and backend reachability; comparing only CPU % without request control is meaningless.

30. Perform single‑instance gray restart

sudo systemctl restart <ServiceName>
sleep 20
sudo systemctl is-active --quiet <ServiceName>
sudo journalctl -u <ServiceName> -n 120 --no-pager

Ensure load balancer has drained traffic and a fallback is ready.

31. Restore probe configuration and restart

sudo cp -a /opt/<AgentPath>/<BackupFile> /opt/<AgentPath>/<ConfigFile>
sudo systemctl restart <ServiceName>
sudo journalctl -u <ServiceName> -n 80 --no-pager

Validate health checks, error rate, trace delivery and JVM metrics; configuration changes without JVM restart do not undo bytecode enhancement.

Selection and Acceptance

SkyWalking suits topology‑centric, scalable backends; Pinpoint excels at detailed Java call traces and existing Pinpoint ecosystems. Neither should be fully injected into the same JVM as a default. Conduct a gray‑scale test with a single probe, compare CPU, RSS, GC pause, P95/P99 latency, error rate, trace completeness and backend report failures under identical load, then decide sampling, plugins and profile scope.