Master Linux Redirection, Pipes, and Text Processing Commands

Redirection

Standard input (stdin) uses exit code 0 with < and <<, standard output (stdout) uses exit code 1 with > or >>, and standard error (stderr) uses exit code 2 with 2> or 2>>. To write both stdout and stderr to one file, use 2>&1.

# Redirect the result of ll to out.txt, overwriting if the file exists
ll /home > out.txt

# Append the result of ll to out.txt
ll /etc >> out.txt

# Write both stdout and stderr to the same file
find /home -name .bashrc > out.txt 2>&1 # note 2>&1 at the end
find /home -name .bashrc &> out.txt # or use &>

Pipe

Use the syntax command A | command B | command C to pass the standard output of command A as the standard input of command B (only commands that can read stdin, such as less, more, head, tail, are suitable; commands like ls, cp, mv cannot be used). To forward stdout as stdin, use 2>&1 to convert stdout to stdin.

tee command

The syntax tee [OPTION]... [FILE]... reads from stdin and writes to both stdout and the specified file(s).

# Show ll results on screen and record to a file
ll /home | tee list_home.out

# Show find results (including errors) on screen and record to a file
find /home -name .bashrc 2>&1 | tee find.out

xargs command

The syntax xargs [options] [command [initial-arguments]] reads stdin, splits it by spaces or newlines, and passes the pieces as arguments to the specified command.

# Pass find results as arguments to ls -lh
find /usr/sbin -perm /7000 | xargs ls -lh

# Pass find results as arguments to du
find /home -name "*.go" | xargs du -cb

Text processing - vim, grep, awk, sed, sort, wc, uniq, tr

grep

Search for lines matching a pattern in a file.

# Find lines containing "rvs" in list.out
grep rvs list.out
# Use a pipe to find lines containing a string in the output of a previous command
ps -ef | grep postgres

cut

Extract sections from each line of a file based on byte, character, or field positions. Use -b, -c, or -f options; if no file is specified, cut reads from stdin.

# Get the first element using ':' as delimiter
echo $PATH | cut -d ':' -f 1
# Remove the first 11 characters of each line from export output
export | cut -c 12-

awk

Usage 1: Process each line with a pattern-action statement.

# Print the 1st and 4th fields of each line
awk '{print $1,$4}' log.txt

Usage 2: Specify field separator with -F.

# Use comma as field separator and print fields 1 and 4
awk -F, '{print $1,$4}' log.txt

sed

Stream editor for applying scripted edits to text files. Common actions include a (append), c (change), d (delete), i (insert), p (print), and s (substitute).

# Append a line after line 4
sed -e 4a"
aaaa" testfile
# Delete lines 2 to 5
nl testfile | sed '2,5d'

sort

Sort the lines of a text file.

# Sort the contents of testfile
sort testfile

wc

Count lines, words, and bytes in a file or from stdin.

# Count the number of lines containing "postgres"
ps -ef | grep postgres | wc -l

uniq

Report or omit repeated lines; typically used after sort.

# Remove duplicate lines from testfile
uniq testfile

tr

Translate or delete characters from stdin.

# Convert lowercase to uppercase
cat testfile | tr a-z A-Z