Essential Linux and Java Debugging Tools Every Engineer Should Know

Linux Command Tools

tail

Most commonly used tail -f to follow file updates.

tail -300f shopbase.log # show last 300 lines and follow

grep

grep forest f.txt      # file search
grep forest f.txt cpf.txt   # multi‑file search
grep 'log' /home/admin -r -n   # recursive search
cat f.txt | grep -i shopbase
grep 'shopbase' /home/admin -r -n --include *.{vm,java}   # include specific extensions
grep 'shopbase' /home/admin -r -n --exclude *.{vm,java}   # exclude specific extensions
seq 10 | grep 5 -A 3   # after match
seq 10 | grep 5 -B 3   # before match
seq 10 | grep 5 -C 3   # context
cat f.txt | grep -c 'SHOPBASE'

awk

Basic usage and field printing.

awk '{print $4,$6}' f.txt
awk '{print NR,$0}' f.txt cpf.txt
awk '{print FNR,$0}' f.txt cpf.txt
awk '{print FNR,FILENAME,$0}' f.txt cpf.txt
awk '{print FILENAME,"NR="NR,"FNR="FNR,"$"NF"="$NF}' f.txt cpf.txt
echo 1:2:3:4 | awk -F: '{print $1,$2,$3,$4}'

Pattern matching examples.

awk '/ldb/ {print}' f.txt   # match ldb
awk '!/ldb/ {print}' f.txt   # not match ldb
awk '/ldb/ && /LISTEN/ {print}' f.txt   # match both
awk '$5 ~ /ldb/ {print}' f.txt   # match column 5

find

sudo -u admin find /home/admin /tmp /usr -name *.log
find . -iname *.txt   # case‑insensitive
find . -type d   # directories only
find /usr -type l   # symbolic links
find /usr -type l -name "z*" -ls   # link details
find /home/admin -size +250000k   # larger than 250 MB
find /home/admin -perm 777 -exec ls -l {} \;   # permission filter
find /home/admin -atime -1   # accessed within 1 day
find /home/admin -ctime -1   # status changed within 1 day
find /home/admin -mtime -1   # modified within 1 day
find /home/admin -amin -1   # accessed within 1 minute
find /home/admin -cmin -1   # status changed within 1 minute
find /home/admin -mmin -1   # modified within 1 minute

pgm

Batch query example.

pgm -A -f vm-shopbase 'cat /home/admin/shopbase/logs/shopbase.log.2017-01-17|grep 2069861630'

tsar

Company‑specific data collection tool.

tsar   # view yesterday's metrics
tsar --live   # real‑time metrics (default 5 s refresh)
tsar -d 20161218   # query a specific day (up to 4 months retained)
tsar --memtsar --loadtsar --cpu   # query single metric with -d

Troubleshooting Tools

btrace

Production‑grade bytecode injection for tracing.

@OnMethod(clazz = "java.util.ArrayList", method="add", location = @Location(value = Kind.CALL, clazz = "/.*/", method = "/.*/"))
public static void m(@ProbeClassName String probeClass, @ProbeMethodName String probeMethod, @TargetInstance Object instance, @TargetMethodOrField String method) {
    if(getInt(field("java.util.ArrayList", "size"), instance) > 479){
        println("check who ArrayList.add method:" + probeClass + "#" + probeMethod + ", method:" + method + ", size:" + getInt(field("java.util.ArrayList", "size"), instance));
        jstack();
        println();
        println("===========================");
        println();
    }
}

Monitor method return values and parameters.

@OnMethod(clazz = "com.taobao.sellerhome.transfer.biz.impl.C2CApplyerServiceImpl", method="nav", location = @Location(value = Kind.RETURN))
public static void mt(long userId, int current, int relation, String check, String redirectUrl, @Return AnyType result) {
    println("parameter# userId:" + userId + ", current:" + current + ", relation:" + relation + ", check:" + check + ", redirectUrl:" + redirectUrl + ", result:" + result);
}

Greys

Java diagnostic tool with features overlapping btrace.

sc -df xxx   # show class details, source location, classloader
trace class method   # method‑level timing breakdown

Related tool: Arthas

javOSize

Allows on‑the‑fly bytecode modification, e.g., inserting logging.

classes   # modify class content at runtime

Arthas

Alibaba open‑source troubleshooting utility.

Repository: https://github.com/alibaba/arthas

JProfiler

Traditional profiling tool, now largely superseded by Greys and btrace for production use.

Website: https://www.ej-technologies.com/products/jprofiler/overview.html

eclipseMAT

Memory analyzer, available as Eclipse plugin or standalone.

Website: http://www.eclipse.org/mat/

zprofiler

Widely used within the company for profiling.

Website: zprofiler.alibaba-inc.com

Java Core Utilities

jps

sudo -u admin /opt/taobao/java/bin/jps -mlvV

jstack

Standard stack trace.

sudo -u admin /opt/taobao/install/ajdk-8_1_1_fp1-b52/bin/jstack 2815
sudo -u admin /opt/taobao/install/ajdk-8_1_1_fp1-b52/bin/jstack -m 2815   # native + java

jinfo

sudo -u admin /opt/taobao/install/ajdk-8_1_1_fp1-b52/bin/jinfo -flags 2815

jmap

Heap inspection and dump.

sudo -u admin /opt/taobao/install/ajdk-8_1_1_fp1-b52/bin/jmap -heap 2815
sudo -u admin /opt/taobao/install/ajdk-8_1_1_fp1-b52/bin/jmap -dump:live,format=b,file=/tmp/heap2.bin 2815
sudo -u admin /opt/taobao/install/ajdk-8_1_1_fp1-b52/bin/jmap -histo 2815 | head -10

jstat

sudo -u admin /opt/taobao/install/ajdk-8_1_1_fp1-b52/bin/jstat -gcutil 2815 1000

jdb

sudo -u admin /opt/taobao/java/bin/jdb -attach 8000

CHLSDB

sudo -u admin /opt/taobao/java/bin/java -classpath /opt/taobao/java/lib/sa-jdi.jar sun.jvm.hotspot.CLHSDB

IntelliJ IDEA Plugins

Key Promoter

Helps you learn keyboard shortcuts by prompting when you use the mouse.

Maven Helper

Analyzes Maven dependencies.

VM Options

Trace class loading: -XX:+TraceClassLoading shows which JAR a class was loaded from.

Generate heap dump on OOM:

-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/home/admin/logs/java.hprof

Jar Conflict Resolution

List all dependencies: mvn dependency:tree > ~/dependency.txt Show specific artifact dependencies: mvn dependency:tree -Dverbose -Dincludes=groupId:artifactId Enable class loading trace in startup scripts: -XX:+TraceClassLoading or -verbose Greys command to locate class source: greys:sc Tomcat classloader locate API:

curl http://localhost:8006/classloader/locate?class=org.apache.xerces.xs.XSObject

List loaded JARs:

curl http://localhost:8006/classloader/jars

Additional Useful Commands

Check active network connections and filter CLOSE_WAIT: netstat -nat|awk '{print $6}'|sort|uniq -c|sort -rn Search OOM killer logs: sudo dmesg|grep -i kill|less Convert dmesg timestamps to real time using epoch calculations.