Master Linux Shell Commands for Fast Data Exploration

01 Shell Commands

Linux dominates with its powerful command line, and shell commands are essential tools for data geeks. Exploratory data analysis often occurs when requirements and data are unclear; using various commands allows exploration and mining.

Data engineers frequently work with Linux. Simple statistics can be performed with wc, sort, uniq, etc. Example:

$ cat file.txt
yunjie
yunjie-talk
yunjie-yun
yunjie
yunjie-shuo
$ sort file.txt | uniq -c | sort -nr | head -5
2 yunjie
1 yunjie-shuo
1 yunjie-talk
1 yunjie-yun

The steps are: cat to view, sort to group identical lines, uniq -c to count, sort -nr to order by count, and head to show the top five.

SQL equivalent:

select word, count(1) cnt
from file
group by word
order by cnt desc
limit 5;

02 Exploratory Analysis

In log analysis, goals may be vague. After obtaining log files, you need to inspect format, compression type, size, and line structure. Common commands include:

gzip/tar – compression/decompression

cat/zcat – file viewing

less/more – paginated viewing (less also handles gzipped files)

head/tail – view first/last lines

wc – count lines, words, characters

du – display disk usage

The philosophy “less is more” illustrates that a simpler tool can be more powerful.

To extract fields, search, modify, or batch‑process files, use commands such as awk, join / cut / paste, fgrep / grep / egrep, find, sed, split, and rename:

# Decompress log
gzip -d a.gz
# View compressed log directly
less a.gz
# Search with context
fgrep 'yunjie-talk' -A 3 -B 3 log.txt
# Find a string in all .log files
find . -name "*.log" | xargs fgrep "hacked by"

Differences: fgrep matches literal strings, grep uses basic regex, and egrep (or grep -E) supports extended regex.

03 Other Useful Commands

Handling file encodings, date operations, file comparisons, network requests, and more:

# Date examples
date -d today +%Y%m%d   # 20160320
date -d yesterday +%Y%m%d   # 20160319
# Unix timestamp
date +%s   # 1458484275
date -d @1458484275   # Sun Mar 20 22:31:15 CST 2016
# Compare two files, show lines only in a.txt
sort a.txt > a.txt.sort
sort b.txt > b.txt.sort
comm -2 -3 a.txt.sort b.txt.sort

04 Batch Operations

After exploration, batch processing becomes essential. Shell scripts enable automation using control structures:

# Conditional directory creation
if [ -d ${base_d} ]; then mkdir -p ${base_d}; fi
# Simple while loop
while true; do do_something; done
# Process all compressed logs
for x in *.log.gz; do gzip -d ${x}; done
# Generate past 8 days' dates
for num in `seq 8 -1 1`; do dd=`date --date="${num} day ago" +%Y%m%d`; echo ${dd}; done

Long‑running tasks are best run with nohup.

05 Conclusion

The commands listed represent only a fraction of the powerful Linux shell toolbox for data analysis. Mastering and combining them into scripts greatly enhances a data engineer’s ability to explore, process, and automate data workflows.