Essential Linux and Java Debugging Commands for Rapid Issue Diagnosis

Linux command utilities

tail

tail -300f shopbase.log   # show last 300 lines and follow file

grep

grep forest f.txt                     # search a single file
grep forest f.txt cpf.txt            # search multiple files
grep 'log' /home/admin -r -n          # recursive search for a keyword
cat f.txt | grep -i shopbase          # case‑insensitive search
grep 'shopbase' /home/admin -r -n --include *.{vm,java}   # limit by extension
grep 'shopbase' /home/admin -r -n --exclude *.{vm,java}    # exclude by extension
seq 10 | grep 5 -A 3   # show 3 lines after match
seq 10 | grep 5 -B 3   # show 3 lines before match
seq 10 | grep 5 -C 3   # show 3 lines around match
cat f.txt | grep -c 'SHOPBASE'   # count occurrences

awk

awk '{print $4,$6}' f.txt
awk '{print NR,$0}' f.txt cpf.txt
awk '{print FNR,$0}' f.txt cpf.txt
awk '{print FNR,FILENAME,$0}' f.txt cpf.txt
awk '{print FILENAME,"NR="NR,"FNR="FNR,"$"NF"=" $NF}' f.txt cpf.txt
echo 1:2:3:4 | awk -F: '{print $1,$2,$3,$4}'

Built‑in variables:

NR : total record number (lines read)

FNR : record number within the current file

NF : number of fields in the current record

find

sudo -u admin find /home/admin /tmp /usr -name *.log          # search multiple directories
find . -iname *.txt                                          # case‑insensitive name match
find . -type d                                               # list sub‑directories
find /usr -type l                                             # list symbolic links
find /usr -type l -name "z*" -ls                             # detailed info for matching links
find /home/admin -size +250000k                               # files larger than 250 GB
find /home/admin -perm 777 -exec ls -l {} ;                 # files with specific permissions
find /home/admin -atime -1                                   # accessed within last day
find /home/admin -ctime -1                                   # status changed within last day
find /home/admin -mtime -1                                   # modified within last day
find /home/admin -amin -1                                    # accessed within last minute
find /home/admin -cmin -1                                    # status changed within last minute
find /home/admin -mmin -1                                    # modified within last minute

pgm

pgm -A -f vm-shopbase 'cat /home/admin/shopbase/logs/shopbase.log.2017-01-17|grep 2069861630'

tsar (internal Alibaba collection tool)

tsar               # show yesterday's metrics
tsar --live        # real‑time metrics (refresh every 5 s)
tsar -d 20161218  # query a specific day (up to ~4 months)
tsar --mem
tsar --load
tsar --cpu

Typical tsar output is shown in the accompanying images.

top & ps

ps -ef | grep java
top -H -p 12345   # replace 12345 with the thread ID (decimal)

Convert thread IDs from decimal to hexadecimal and use jstack for stack traces.

netstat + awk

netstat -nat | awk '{print $6}' | sort | uniq -c | sort -rn

Shows current connections and highlights high CLOSE_WAIT counts.

Java troubleshooting tools

btrace

Production‑grade bytecode‑injection tool for tracing method calls and monitoring parameters. Example use cases:

Identify which threads invoke ArrayList.add when the list size exceeds 500.

Monitor method return values and request arguments.

Repository: https://github.com/btraceio/btrace

Note: btrace 1.3.9 output can be unstable; multiple triggers may be needed. Regex patterns must be scoped carefully to avoid CPU spikes. Because btrace works via bytecode injection, a JVM restart is required to restore normal operation.

Greys

Another bytecode‑instrumentation tool with overlapping features: sc -df xxx – show class details, source location, and classloader hierarchy. trace class method – print method‑level execution time, similar to JProfiler.

Related projects: http://www.atatech.org/articles/26247 http://mw.alibaba-inc.com/products/arthas/docs/middleware-container/arthas.wiki/home.html

javOSize

Allows on‑the‑fly bytecode modification to inject logging statements, useful for immediate visibility into class behavior.

JProfiler

Historically used for deep performance analysis; now often replaced by Greys or btrace for production environments.

Eclipse MAT

Memory Analyzer Tool, available as an Eclipse plugin or standalone application. URL: http://www.eclipse.org/mat/

zprofiler

Internal Alibaba profiling tool; widely used within the organization.

jps

sudo -u admin /opt/taobao/java/bin/jps -mlvV

jstack

sudo -u admin /opt/taobao/install/ajdk-8_1_1_fp1-b52/bin/jstack 2815          # standard stack trace
sudo -u admin /opt/taobao/install/ajdk-8_1_1_fp1-b52/bin/jstack -m 2815   # native + Java stack

jinfo

sudo -u admin /opt/taobao/install/ajdk-8_1_1_fp1-b52/bin/jinfo -flags 2815

jmap

sudo -u admin /opt/taobao/install/ajdk-8_1_1_fp1-b52/bin/jmap -heap 2815               # heap summary
sudo -u admin /opt/taobao/install/ajdk-8_1_1_fp1-b52/bin/jmap -dump:live,format=b,file=/tmp/heap2.bin 2815   # live heap dump
sudo -u admin /opt/taobao/install/ajdk-8_1_1_fp1-b52/bin/jmap -histo 2815 | head -10   # top object histogram

Typical jmap output is illustrated in the images.

jstat

sudo -u admin /opt/taobao/install/ajdk-8_1_1_fp1-b52/bin/jstat -gcutil 2815 1000

jdb

sudo -u admin /opt/taobao/java/bin/jdb -attach 8000

Used for remote debugging of pre‑release Java services.

CHLSDB

sudo -u admin /opt/taobao/java/bin/java -classpath /opt/taobao/java/lib/sa-jdi.jar sun.jvm.hotspot.CLHSDB

Provides low‑level JVM inspection similar to jstack and jmap.

IntelliJ IDEA plugins

Key Promoter

Shows the keyboard shortcut after a mouse action, helping users learn shortcuts.

Maven Helper

Analyzes Maven dependencies to resolve conflicts.

VM options tips

Find which JAR a class was loaded from: -XX:+TraceClassLoading Enable heap dump on OOM:

-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/home/admin/logs/java.hprof

Jar conflict resolution

List all dependencies: mvn dependency:tree > ~/dependency.txt Show a specific group/artifact: mvn dependency:tree -Dverbose -Dincludes=groupId:artifactId Use -XX:+TraceClassLoading or -verbose in JVM start scripts.

Greys command sc also reveals class loading locations.

ALI‑TOMCAT utilities

List container‑loaded JARs: curl http://localhost:8006/classloader/jars Locate the JAR providing a specific class:

curl http://localhost:8006/classloader/locate?class=org.apache.xerces.xs.XSObject

Images illustrate the HTTP responses.

OOM Killer analysis

When a Java process disappears, the kernel log can reveal OOM‑killer activity: sudo dmesg | grep -i kill | less Typical snippet:

[6710782.021013] java invoked oom-killer: gfp_mask=0xd0, order=0, oom_adj=0, oom_score_adj=0
[6710784.698347] Memory cgroup out of memory: Kill process 215701 (java) score 854 ...

The OOM killer selects the process with the highest score to free memory. The log timestamp can be converted to a human‑readable date with:

date -d "1970-01-01 UTC $(($(date +%s) - $(cat /proc/uptime | cut -d' ' -f1) + 12288812.926194)) seconds"

RateLimiter (Guava)

For fine‑grained QPS control, Guava's RateLimiter can enforce limits such as 400 requests per second. Example usage:

RateLimiter limiter = RateLimiter.create(400.0); // 400 permits per second
limiter.acquire(); // blocks until a permit is available