Essential Linux and Java Tools for Fast Troubleshooting and Performance Tuning

Linux Command Tools

tail

Most commonly used: tail -300f shopbase.log # show last 300 lines and follow live.

grep

grep forest f.txt            # file search
grep forest f.txt cpf.txt    # multi‑file search
grep 'log' /home/admin -r -n # recursive search
cat f.txt | grep -i shopbase
grep 'shopbase' /home/admin -r -n --include *.{vm,java}   # include specific extensions
grep 'shopbase' /home/admin -r -n --exclude *.{vm,java}   # exclude specific extensions
seq 10 | grep 5 -A 3      # after match
seq 10 | grep 5 -B 3      # before match
seq 10 | grep 5 -C 3      # context
cat f.txt | grep -c 'SHOPBASE'

awk

Basic commands:

awk '{print $4,$6}' f.txt
awk '{print NR,$0}' f.txt cpf.txt
awk '{print FNR,$0}' f.txt cpf.txt
awk '{print FNR,FILENAME,$0}' f.txt cpf.txt
awk '{print FILENAME,"NR="NR,"FNR="FNR,"$"NF"="$NF}' f.txt cpf.txt
echo 1:2:3:4 | awk -F: '{print $1,$2,$3,$4}'

Matching examples:

awk '/ldb/ {print}' f.txt               # match ldb
awk '!/ldb/ {print}' f.txt              # not match ldb
awk '/ldb/ && /LISTEN/ {print}' f.txt   # match both
awk '$5 ~ /ldb/ {print}' f.txt          # match column 5

find

sudo -u admin find /home/admin /tmp /usr -name *.log
find . -iname *.txt
find . -type d
find /usr -type l
find /usr -type l -name "z*" -ls
find /home/admin -size +250000k
find /home/admin -perm 777 -exec ls -l {} \;
find /home/admin -atime -1
find /home/admin -ctime -1
find /home/admin -mtime -1
find /home/admin -amin -1
find /home/admin -cmin -1
find /home/admin -mmin -1

pgm

Batch query example:

pgm -A -f vm-shopbase 'cat /home/admin/shopbase/logs/shopbase.log.2017-01-17|grep 2069861630'

tsar

Company‑specific data collection tool:

tsar   # view recent day metrics
tsar --live   # real‑time metrics (default 5 s refresh)
tsar -d 20161218   # query a specific day (up to ~4 months)
tsar --memtsar --loadtsar --cpu   # query single metric for a day

Powerful Troubleshooting Tools

btrace

Bytecode‑injection based live tracing. Example to monitor ArrayList.add when size > 500:

@OnMethod(clazz = "java.util.ArrayList", method="add", location = @Location(value = Kind.CALL, clazz = "/.*/", method = "/.*/"))
public static void m(@ProbeClassName String probeClass, @ProbeMethodName String probeMethod, @TargetInstance Object instance, @TargetMethodOrField String method) {
    if (getInt(field("java.util.ArrayList", "size"), instance) > 479) {
        println("check who ArrayList.add method:" + probeClass + "#" + probeMethod + ", method:" + method + ", size:" + getInt(field("java.util.ArrayList", "size"), instance));
        jstack();
        println();
        println("===========================");
        println();
    }
}

Monitor method parameters and return value:

@OnMethod(clazz = "com.taobao.sellerhome.transfer.biz.impl.C2CApplyerServiceImpl", method="nav", location = @Location(value = Kind.RETURN))
public static void mt(long userId, int current, int relation, String check, String redirectUrl, @Return AnyType result) {
    println("parameter# userId:" + userId + ", current:" + current + ", relation:" + relation + ", check:" + check + ", redirectUrl:" + redirectUrl + ", result:" + result);
}

Greys

Features overlapping with btrace, e.g.:

sc -df xxx   # show class details, source location, classloader
trace class method   # print method call timings

Related tool: Arthas

javOSize

Provides a classes command to modify bytecode on‑the‑fly for quick logging.

JProfiler

Previously used for many issues; now often replaced by Greys and btrace.

Major Debugging Tools

eclipse MAT

Memory analysis tool, available as an Eclipse plugin or standalone.

zprofiler

Widely used within the company for deep performance analysis.

Java Core Utilities

jps

sudo -u admin /opt/taobao/java/bin/jps -mlvV

jstack

sudo -u admin /opt/taobao/install/ajdk-8_1_1_fp1-b52/bin/jstack 2815
sudo -u admin /opt/taobao/install/ajdk-8_1_1_fp1-b52/bin/jstack -m 2815

jinfo

sudo -u admin /opt/taobao/install/ajdk-8_1_1_fp1-b52/bin/jinfo -flags 2815

jmap

Heap inspection and dump:

sudo -u admin /opt/taobao/install/ajdk-8_1_1_fp1-b52/bin/jmap -heap 2815
sudo -u admin /opt/taobao/install/ajdk-8_1_1_fp1-b52/bin/jmap -dump:live,format=b,file=/tmp/heap2.bin 2815
sudo -u admin /opt/taobao/install/ajdk-8_1_1_fp1-b52/bin/jmap -histo 2815 | head -10

jstat

sudo -u admin /opt/taobao/install/ajdk-8_1_1_fp1-b52/bin/jstat -gcutil 2815 1000

jdb

sudo -u admin /opt/taobao/java/bin/jdb -attach 8000

CHLSDB

sudo -u admin /opt/taobao/java/bin/java -classpath /opt/taobao/java/lib/sa-jdi.jar sun.jvm.hotspot.CLHSDB

IntelliJ IDEA Plugins

Key Promoter

Helps memorize shortcut keys.

Maven Helper

Analyzes Maven dependencies.

VM Options

-XX:+TraceClassLoading   # show which file loaded a class
-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/home/admin/logs/java.hprof

Jar Conflict Resolution

Common commands:

mvn dependency:tree > ~/dependency.txt
mvn dependency:tree -Dverbose -Dincludes=groupId:artifactId
-XX:+TraceClassLoading   # add to startup script
-verbose                 # add to startup script
greys:sc                 # show class loading location

ALI‑TOMCAT Utilities

List loaded jars and locate class sources via HTTP endpoints.

Other Useful Commands

dmesg

Detect OOM‑killer events:

sudo dmesg | grep -i kill | less

Convert dmesg timestamps to real time using the provided formula.