Master the Command Line: Essential Tips and Tricks for Every Engineer

The Art of Command Line is a GitHub open‑source project that has topped the weekly GitHub TOP list with 53,972 stars.

GitHub repository: https://github.com/jlevy/the-art-of-command-line

The following is the Chinese README (zh.md) of the project; feel free to check it out.

Command Line Art

Preface

Basics

Daily Use

File and Data Processing

System Debugging

One‑liners

Obscure but Useful

OS X Only

Windows Only

More Resources

Disclaimer

Proficiency with the command line is often overlooked or considered hard to master, but it actually increases an engineer's flexibility and productivity. This article is a summary of useful command‑line tricks discovered while working on Linux. Some tricks are very basic, others are complex and even obscure. The article is not long, but mastering all the listed tricks will teach you a lot about the command line.

This article is a collaborative effort of many authors and translators. Some content first appeared on Quora, later migrated to GitHub and improved by many experts. If you find errors or improvements, please contribute.

1. Preface

The scope covers both beginners and experienced users. The article aims for broad coverage, concrete examples, and conciseness, avoiding redundant or easily searchable details. In specific scenarios it serves as basic knowledge or can save you a lot of time.

The article is mainly written for Linux, but OS X‑only and Windows‑only sections contain relevant commands for those platforms. Apart from these sections, most content works on other Unix‑like systems, OS X, and even Cygwin.

The focus is on interactive Bash, though many tricks apply to other shells and Bash scripts.

In addition to standard Unix commands, the article includes some commands that depend on specific packages, provided they are valuable.

Note: To keep the article within one page, some detailed information can be found in the referenced pages. Use Google or another search engine to look up more details. Some commands require package managers such as apt-get, yum, dnf, pacman, pip, or brew to install required programs.

If you encounter problems, try Explainshell to get explanations for commands, arguments, and pipelines.

2. Basics

Read the Bash manual: man bash. It is short and easy to understand. Other shells are useful, but Bash is powerful and almost always available.

Familiarize yourself with at least one text editor. Vim ( vi) is often the best choice for terminal editing.

Learn to use man, apropos, help, help -d, and type to get documentation and determine whether a command is an executable, a built‑in, or an alias.

Master redirection: >, >>, |, and understand stdout vs. stderr.

Learn wildcard usage: *, ?, [...], and the differences between quoting with ' and ".

Get comfortable with Bash job control: &, ctrl‑z, ctrl‑c, jobs, fg, bg, kill.

Use ssh for remote login and ssh‑agent / ssh‑add for password‑less authentication.

Basic file management commands: ls, ls -l, less, head, tail, tail -f, ln, ln -s, chown, chmod, du, df, mount, fdisk, mkfs, lsblk. Understand inodes with ls -i and df -i.

Network tools: ip or ifconfig, dig.

Version control: git.

Regular expressions: grep / egrep with options -i, -o, -v, -A, -B, -C.

Use apt-get, yum, dnf, or pacman to install packages, and pip for Python tools.

3. Daily Use

Tab completes arguments; ctrl‑r searches command history. ctrl‑w deletes the last word, ctrl‑u deletes to the beginning of the line, alt‑b / alt‑f move by word, ctrl‑a / ctrl‑e move to line start/end, ctrl‑k deletes to line end, ctrl‑l clears the screen. See man readline for more shortcuts.

Switch to vi‑style key bindings with set -o vi or back to emacs with set -o emacs.

Press ctrl‑x ctrl‑e (or escape‑v in vi mode) to open the default editor for the current command line.

View command history with history, re‑execute with !n, use !$ for the last argument, !! for the previous command, or use ctrl‑r / alt‑. for similar functionality.

Navigate directories with cd, cd ~, cd -, and use $HOME in scripts.

Use xargs (or parallel) for batch processing; control batch size with -L and parallelism with -P. Test with xargs echo.

Show process trees with pstree -p.

Search processes with pgrep and send signals with pkill (e.g., -f for full‑command match).

Stop a process with kill -STOP [pid] and view signal list with man 7 signal.

Run background jobs persistently with nohup or disown.

Check listening ports with netstat -lntp or ss -plat (add -u for UDP).

List open sockets/files with lsof.

Show system uptime with uptime or w.

Create command aliases, e.g., alias ll='ls -latr'.

Store aliases, shell options, and functions in ~/.bashrc; store login commands in ~/.bash_profile (different files for GUI shells and cron).

Synchronize configuration files across machines with Git.

When filenames contain spaces, quote variables: "$FOO". Use -0 or --print0 with find / xargs to handle NUL‑terminated names.

In Bash scripts, enable strict mode with set -euo pipefail and use trap for ERR and EXIT handling.

Run commands in a subshell with parentheses, e.g., (cd /some/dir && other‑command).

Variable expansions: ${name:?error}, default values with ${name:-default}, arithmetic with $((...)), sequences with {1..10}, substring removal with ${var%suffix} and ${var#prefix}.

Brace expansion for concise file operations, e.g., mv foo.{txt,pdf} some‑dir or mkdir -p test-{a,b,c}/subtest-{1,2,3}.

Process substitution: diff /etc/hosts <(ssh host cat /etc/hosts).

Here documents: cat <<EOF ... EOF.

Redirect both stdout and stderr: some‑command >logfile 2>&1 or some‑command &>logfile. Append </dev/null to avoid hanging input.

View ASCII tables with man ascii, man unicode, man utf‑8, man latin1.

Use terminal multiplexers screen or tmux; byobu adds more features; dtach is a lightweight alternative.

SSH tunneling with -L, -D, or -R for remote web access.

SSH config optimizations (e.g., TCPKeepAlive=yes, ServerAliveInterval=15, Compression=yes, ControlMaster auto, etc.). Use StrictHostKeyChecking=no and ForwardAgent=yes only on trusted networks.

Consider mosh as a UDP‑based SSH replacement for better resilience.

Get octal file permissions with stat -c '%A %a %n' /etc/timezone.

Change file attributes with chattr (e.g., sudo chattr +i /critical/file).

Save and restore ACLs with getfacl -R /some/path > permissions.txt and setfacl --restore=permissions.txt.

Create empty files efficiently with truncate, fallocate, xfs_mkfile, or mkfile.

4. File and Data Processing

Find files by name in the current directory: find . -iname '*something*'. Search the whole system with locate something (remember to run updatedb to refresh the index).

Search source code with ag (faster than grep -r).

Convert HTML to plain text with lynx -dump -stdin.

Convert between markup formats with pandoc.

Process XML with xmlstarlet.

Handle JSON with jq.

Process YAML with shyaml.

CSV/Excel tools: in2csv, csvcut, csvjoin, csvgrep.

AWS tools: s3cmd, s4cmd, aws, saws.

Sort and deduplicate with sort and uniq (including -u, -d).

Understand LC_ALL and locale effects on sorting.

Use cut, paste, join for column manipulation.

Count lines/words/bytes with wc ( -l, -m, -w, -c).

Duplicate output with tee.

Statistical analysis with datamash.

Be aware of locale impact on sort; use export LC_ALL=C for byte‑wise order.

Set temporary environment variables for a single command, e.g., TZ=Pacific/Fiji date.

Use awk and sed for simple data processing.

Replace strings in many files with perl -pi.bak -e 's/old-string/new-string/g' *.txt or rename / repren.

Synchronize files with rsync (fast, can resume, delete‑only mode).

Show copy progress with pv, pycp, progress, or rsync --progress. For block copies, use dd status=progress.

Randomly shuffle lines with shuf.

Sort numerically with -n or human‑readable sizes with -h. Use -k and -t for field‑based sorting; stable sort with sort -s.

Insert a literal tab character with ctrl‑v [Tab] or $'\t'.

Diff and patch tools: diff, patch, diffstat, diff -r, vimdiff.

Binary inspection with hd, hexdump, xxd, edit with bvi, hexedit, biew. Search strings with strings and grep.

Create binary deltas with xdelta3.

Change text encoding with iconv or advanced Unicode conversion with uconv.

Split files by size with split or by pattern with csplit.

Date utilities: dateutils suite ( dateadd, datediff, strptime).

Compress‑aware tools: zless, zmore, zcat, zgrep.

File attributes with chattr (e.g., immutable flag).

Save and restore ACLs with getfacl and setfacl.

Create empty files efficiently with truncate, fallocate, xfs_mkfile, mkfile.

5. System Debugging

Web debugging with curl, curl -I, wget, or modern httpie.

Monitor CPU and disks with top ( htop), iostat, iotop, or iostat -mxz 15.

Inspect network connections with netstat and ss.

Overall system view with dstat or richer UI with glances.

Memory status with free and vmstat (note the “cached” column).

Java debugging: kill -3 <pid> for stack trace, jps, jstat, jstack, jmap, SJK tools.

Trace routes with mtr.

Network packet capture with wireshark, tshark, ngrep.

System call tracing with strace and library call tracing with ltrace (use -c for profiling, -p to attach).

Inspect shared libraries with ldd (avoid untrusted files).

Use /proc for low‑level info: /proc/cpuinfo, /proc/meminfo, /proc/cmdline, /proc/<pid>/cwd, /proc/<pid>/exe, /proc/<pid>/fd/, /proc/<pid>/smaps.

Detect deleted files still held open with lsof | grep deleted.

Historical performance data with sar.

Deep performance analysis with stap, perf, sysdig.

Identify OS version with uname, uname -a, or lsb_release -a.

Check kernel messages with dmesg.

6. One‑liners

Set operations on files using sort and uniq (union, intersection, difference).

Search all files with grep . * or preview first 100 lines with head -100 *.

Sum the third column of a file with awk '{ x += $3 } END { print x }' myfile.

Recursively list files with size/date: find . -type f -ls.

Count occurrences of a query parameter in logs:

egrep -o 'acct_id=[0-9]+' access.log | cut -d= -f2 | sort | uniq -c | sort -rn

.

Watch directory changes: watch -d -n 2 'ls -rtlh | tail' or monitor network settings: watch -d -n 2 ifconfig.

Randomly pick a tip from the document using a Bash function that fetches the README, converts to HTML, extracts a random list item, and formats it.

7. Obscure but Useful

expr

: evaluate expressions or regex matches. m4: simple macro processor. yes: repeatedly print a string. cal: display a calendar. env: run a command with a modified environment. printenv: display environment variables. look: find words/lines starting with a prefix. cut, paste, join: data manipulation. fmt: reformat paragraphs. pr: format text into pages/columns. fold: wrap long lines. column: align text into columns. expand / unexpand: convert tabs to spaces and back. nl: number lines. seq: print sequences of numbers. bc: calculator. factor: factor numbers. gpg: encrypt and sign files. toe: list terminfo entries. nc: network debugging and data transfer. socat: socket proxy (similar to netcat). slurm: network traffic visualization. dd: low‑level copy between files/devices. file: determine file type. tree: display directory tree. stat: file status. time: measure command execution time. timeout: limit command run time. lockfile: create lock files. logrotate: rotate, compress, and mail logs. watch: repeatedly run a command and highlight changes. when‑changed: run a command when a file changes (see inotifywait, entr). tac: output a file in reverse. shuf: randomly select lines. comm: compare two sorted files line by line. strings: extract printable strings from binaries. tr: translate characters. iconv / uconv: convert text encodings. split / csplit: split files. sponge: soak up input before writing to the same file. units: convert between measurement units. apg: generate random passwords. xz: high‑ratio compression. ldd: list shared library dependencies. nm: list symbols from object files. ab or wrk: benchmark web servers. strace: trace system calls. mtr: improved network traceroute. cssh: visual concurrent shells. rsync: synchronize files/directories locally or over SSH. wireshark / tshark / ngrep: packet capture and analysis. host / dig: DNS lookups. lsof: list open files and ports. dstat: system status overview. glances: high‑level multi‑subsystem overview. iostat: disk I/O statistics. mpstat: CPU usage statistics. vmstat: memory usage statistics. htop: enhanced top. last: login history. w: who is logged in. id: user/group IDs. sar: historical system data. iftop / nethogs: per‑process/network bandwidth. ss: socket statistics. dmesg: kernel and boot messages. sysctl: view/modify kernel parameters at runtime. hdparm: SATA/ATA disk performance. lsblk: list block devices as a tree. lshw, lscpu, lspci, lsusb, dmidecode: hardware information. lsmod / modinfo: kernel modules. fortune, ddate, sl: fun utilities.

8. macOS‑Only

The following tips apply only to macOS.

Use Homebrew ( brew) or MacPorts ( port) for package management to install most of the commands listed above.

Copy command output to the clipboard with pbcopy and paste with pbpaste.

Enable the Option key as Meta in Terminal preferences to use alt‑b, alt‑f, etc.

Open files with GUI apps using open or open -a /Applications/Whatever.app.

Spotlight search via mdfind; retrieve metadata with mdls (e.g., EXIF data).

macOS is BSD‑based; many commands ( ps, ls, tail, awk, sed) differ subtly from Linux. GNU versions may be installed as gawk, gsed, etc. Write portable scripts accordingly.

Use sw_vers to get macOS version information.

9. Windows‑Only

The following tips apply only to Windows.

9.1 Getting Unix Tools on Windows

Install Cygwin to get a full Unix‑like shell on Windows; most of the article’s content then applies.

Use Bash on Ubuntu on Windows (WSL) for a familiar Bash environment with many Unix tools. Note that Linux‑compiled programs run, but native Windows programs do not.

Install MinGW and its MSYS package for GNU development tools (gcc, gawk, make, grep). MSYS lacks some Cygwin features but is useful for native Windows ports.

Cash provides a lightweight Unix‑like environment with limited commands.

9.2 Useful Native Windows Command‑Line Tools

Use wmic to script and execute most Windows system‑administration tasks.

Native networking tools include ping, ipconfig, tracert, and netstat.

Run many Windows tasks via Rundll32.

9.3 Cygwin Tips

Install additional Unix programs via Cygwin's package manager.

Use mintty as your terminal window.

Access the Windows clipboard via /dev/clipboard.

Open files with the default program using cygstart.

Access the Windows registry with regtool.

Windows drive paths (e.g., C: ext) are accessed in Cygwin as /cygdrive/c/text. The Cygwin root / maps to C:\cygwin. Convert paths with cygpath.

Use wmic from Cygwin for Windows system management.

Cash offers a minimal Unix‑like environment on Windows.

MinGW + MSYS provides Bash, gawk, make, grep, etc., useful for native Windows ports.

10. More Resources

awesome‑shell: curated list of command‑line tools and resources – https://github.com/alebcay/awesome-shell

awesome‑osx‑command‑line: deeper guide for macOS command line – https://github.com/herrbischoff/awesome-osx-command-line

Strict mode: write better scripts – http://redsymbol.net/articles/unofficial-bash-strict-mode/

shellcheck: static analysis for shell scripts – https://github.com/koalaman/shellcheck

Filenames and Pathnames in Shell: handling filenames correctly – http://www.dwheeler.com/essays/filenames-in-shell.html

Data Science at the Command Line: tools for data science – http://datascienceatthecommandline.com/#tools

11. Disclaimer

Except for very small tasks, your code should be readable by others. With great power comes responsibility; just because you can perform clever Bash tricks doesn’t mean you should. :)

This article is licensed under the Creative Commons Attribution‑ShareAlike 4.0 International License.