Common Linux Commands for Diagnosing Java Backend Performance Issues

Memory Bottleneck

free

free

displays memory usage, including physical, swap, and kernel buffers. free -h -s 3 updates the output every three seconds.

[1014154@cc69dd4c5-4tdb5 ~]$ free
              total        used        free      shared  buff/cache   available
Mem:      119623656    43052220    45611364     4313760    30960072    70574408
Swap:            0          0          0
[1014154@cc69dd4c5-4tdb5 ~]$ free -h -s 3
              total        used        free      shared  buff/cache   available
Mem:          114G          41G          43G        4.1G          29G          67G
Swap:          0B          0B          0B

Mem

: memory usage. Swap: swap usage. total: total physical memory and swap. used: memory already used. free: truly unused physical memory. shared: shared memory. buff/cache: memory used by buffers and cache. available: memory available to applications (≈ free + buffer + cache).

Swap Space

Swap is a disk area used when physical memory is scarce; it can alleviate memory shortage but has lower performance due to disk I/O.

vmstat (recommended)

vmstat

(Virtual Memory Statistics) monitors overall system performance, including memory, processes, CPU, and I/O. vmstat 5 3 reports statistics every 5 seconds, three times.

[1014154@cc69dd4c5-4tdb5 ~]$ vmstat 5 3
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 8  0      0 45453212 374768 30763728    0    0    14    99    1    1 11 10 78  0  1
10  0      0 45489232 374768 30763360    0    0     2  1275 95118 97908 13 11 75  0  1
 6  0      0 45452908 374768 30765148    0    0     0  3996 89924 92073 12 10 78  0  1

procs

r

: number of processes running or waiting for CPU time slice; a sustained value higher than CPU cores indicates CPU shortage. b: processes waiting for I/O or memory swap.

memory

swpd

: memory swapped to disk (virtual memory). free: current free physical memory. buff: buffer size for block devices. Cache: file system cache.

swap

si

: data read from disk to memory (swap‑in). so: data written from memory to disk (swap‑out).

io

bi

: bytes read from block devices (disk read). bo: bytes written to block devices (disk write).

system

in

: interrupts per second. cs: context switches per second; high values suggest excessive thread/process switching.

CPU

us

: user‑mode CPU usage; >50% long‑term may need code or algorithm optimization. sy: kernel‑mode CPU usage. id: idle CPU percentage. wa: I/O wait percentage; >20% indicates serious I/O wait.

sar

sar -r 3

shows memory statistics every three seconds, similar to free. sar -u 3 displays CPU usage breakdown, focusing on %iowait and %idle for diagnosing I/O or CPU bottlenecks.

CPU Bottleneck

Check CPU Core Count

# Total cores = physical CPUs * cores per CPU
# Total logical CPUs = physical CPUs * cores per CPU * hyper‑threading

CPU Information

[1014154@cc69dd4c5-4tdb5 ~]$ cat /proc/cpuinfo | grep name | cut -f2 -d: | uniq -c
      32  Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz

Physical CPU Count

[1014154@cc69dd4c5-4tdb5 ~]$ cat /proc/cpuinfo| grep "physical id"| sort| uniq| wc -l
16

Core Count per Physical CPU

[1014154@cc69dd4c5-4tdb5 ~]$ cat /proc/cpuinfo| grep "cpu cores"| uniq
cpu cores       : 2

Logical CPU Count

[1014154@cc69dd4c5-4tdb5 ~]$ cat /proc/cpuinfo| grep "processor"| wc -l
32

top

top

shows overall CPU consumption, with fields for user, system, idle, nice, iowait, etc. Keyboard shortcuts: Shift+H shows Java threads, Shift+M sorts by memory, Shift+P sorts by CPU, Shift+T sorts by cumulative CPU, 1 toggles per‑CPU view.

top - 15:24:11 up 8 days, 7:52, 1 user, load average: 5.73, 6.85, 7.33
Tasks: 17 total, 1 running, 16 sleeping, 0 stopped, 0 zombie
%Cpu(s): 13.9 us, 9.2 sy, 0.0 ni, 76.1 id, 0.1 wa, 0.0 hi, 0.1 si, 0.7 st
...

Network Bottleneck

Packet Loss and Errors

Use watch -n 2 more /proc/net/dev to monitor drops and errors on interfaces.

Inter-|   Receive                                                |  Transmit
 face |bytes    packets errs drop fifo frame compressed multicast|bytes    packets errs drop fifo colls carrier compressed
   lo:   10025     130    0    0    0     0          0          0    10025     130    0    0    0     0       0          0
 ens33: 759098071  569661    0    0    0     0          0          0 19335572  225551    0    0    0     0       0          0

Routing Path

traceroute <ip>

shows each hop and latency.

traceroute 14.215.177.38
 1  CD-HZTK5H2.mshome.net (192.168.137.1)  0.126 ms * *
 2  * * *
 3  10.250.112.3 (10.250.112.3)  12.587 ms 12.408 ms 12.317 ms
 ...

Network Errors

netstat -i

displays interface error counters.

Iface   MTU   RX-OK RX-ERR RX-DRP RX-OVR   TX-OK TX-ERR TX-DRP TX-OVR Flg
ens33  1500  570291     0      0     0      225897     0      0      0 BMRU
lo    65536    130     0      0     0        130     0      0      0 LRU

Retransmission Rate

Read /proc/net/snmp and compute tcpetr = RetransSegs / OutSegs.

Tcp: 1 200 120000 -1 376 6 0 0 4 236711 223186 292 0 4 0
Retransmission rate = 292 / 223186 ≈ 0.13%

Disk Bottleneck

Disk Space

[root@localhost ~]$ df -hl
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/centos_aubin-root 27G  5.6G   22G  21% /
/dev/sda1               1014M  211M  804M  21% /boot

Disk Usage

[root@localhost ~]$ du -sh
64K

Overall Disk I/O

[root@localhost ~]$ iostat
avg-cpu:  %user   %nice %system %iowait %steal %idle
          0.17    0.00    0.20    0.46    0.00   99.17
Device:    tps   kB_read/s  kB_wrtn/s  kB_read  kB_wrtn
sda        1.56      30.45      39.61  4659620  6060644

Detailed Disk I/O

[root@localhost ~]$ iostat -x 1 3
Device:   rrqm/s wrqm/s   r/s   w/s  rkB/s  wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda          0.01   0.49 0.63 0.95 30.59 39.78   89.58   0.34 214.23 49.16 323.48 8.55 1.34

Top I/O Processes

Use iotop to locate processes with high disk I/O, optionally filtering by PID with iotop -p <pid>.

[root@localhost ~]$ iotop
TID  PRIO USER   DISK READ  DISK WRITE  SWAPIN  IO>  COMMAND
123931 be/4 root   0.00 B/s   0.00 B/s   0.00%  0.02% [kworker/1:30]
...

Application Bottleneck

Find Process ID

[root@localhost ~]$ ps -ef | grep java
root 124146 1984 0 09:13 pts/0 00:00:06 java -jar arthas-demo.jar

Thread Count

[root@localhost ~]$ ps -efL | grep 124146 | wc -l
12

Thread Details

[root@localhost ~]$ ps -Lp 124146 cu
USER   PID   LWP %CPU NLWP %MEM   VSZ   RSS TTY   STAT START   TIME COMMAND
root 124146 124147 0.0  11 2.5 2489116 35724 pts/0 Sl+ 09:13 0:01 java
...

Detect Deadlocks

[root@localhost ~]$ jstack -l 124146
"C1 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007f27f013c000 nid=0x1e4f9 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE
...

Search Error Logs

find / -type f -name "*.log" | xargs grep "ERROR"
/var/log/tuned/tuned.log:2020-03-13 18:05:59,145 ERROR ...

Summary Table

Category

Command

Description

Notes

Memory Bottleneck

free

Show memory usage

vmstat 5 3

Detailed swap in/out and CPU stats

Recommended

CPU Bottleneck

top -H

Sort by CPU usage per thread

ps -Lp <pid> cu

CPU usage per thread of a process

Network Bottleneck

watch more /proc/net/dev

Monitor packet drops and errors

Focus on drop and total traffic

traceroute <ip>

Show route hops and latency

Disk Bottleneck

iostat -x -k -d 1

Detailed disk read/write stats

Watch %iowait, await, %util

iotop

Identify processes with high I/O

Use after iostat shows I/O pressure

Application Bottleneck

ps -ef | grep java

Find Java process PID

jstack -l <pid>

Check for deadlocks

find / -type f -name "*.log" | xargs grep "ERROR"

Count error lines in all logs

For any Linux command, use --help or man to view detailed usage information.