Mastering Linux Text Processing: grep, sed, and awk Explained with Real Examples

About the Linux "Three Musketeers"

grep – filters keyword information; primarily used to search data within text files.

sed – edits text data and can modify the original file content.

awk – filters file data, extracts fields, and can produce formatted output.

What is sed

sed software itself.

The processing commands provided by sed.

Source data supplied to sed.

sed Syntax Format

sed [options] [sed‑builtin‑command] [input‑file]

Explanation:
1. There must be at least one space between the sed program, its options, commands, and the input file.
2. To avoid confusion, refer to the program as "sed software" and its built‑in commands as "sed‑commands".
3. sed‑commands can be a single command or a combination of multiple commands.
4. The input‑file is optional; sed can also read from standard input or a pipe.

Syntax
sed replace‑character‑data
s replace‑instruction
#pattern#replacement#

sed 's#old_data#new_data#' file.txt

sed Command Execution Process

Overview:
sed reads a line from a file or pipe, processes the line, outputs the line, then repeats the cycle.

sed Parameters

sed modifies data in the pattern space (memory). By default the result is printed but not written to the file.
To write the result back to the file, use the -i option:

sed -i 's#old#new#' file.txt

options:
-n  Suppress default output; often used with the p command.
-e  Allow multiple sed commands on a single line.
-f  Followed by a script file name.
-r  Enable extended regular expressions.
-i  Edit the file in place; without -i changes are only in memory.

sed Commands

a  Append after the addressed line.
 c  Change the addressed line.
 d  Delete the addressed line.
 D  Delete part of the pattern space up to a newline.
 i  Insert before the addressed line.
 h  Copy pattern space to hold space.
 H  Append pattern space to hold space.
 g  Copy hold space to pattern space.
 G  Append hold space to pattern space.
 x  Exchange pattern and hold spaces.
 l  Print non‑printable characters.
 n  Read next line, replace pattern space.
 N  Append next line to pattern space without replacing.
 p  Print pattern space (usually with -n).
 P  Print up to the first newline.
 q  Quit.
 r  Read a file and insert its contents.
 s  Substitute (e.g., s#old#new#g).
 w  Write pattern space to a file.
 y  Translate characters.
 :label  Define a label.
 t  If a substitution succeeded, branch to label.

sed Range Matching

Modifying Characters and Suppressing Default Output

sed Insert, Delete, Update, Query (Practice)

Insert a line after the third line: 3a This is an inserted line Insert a line before the third line:

3i This line is inserted before line 3

Appending Multiple Lines with cat and echo

cat >>my.log <<EOF
Hello
I am fine
EOF

echo -e "hello
world
你好
我也好" > hello.log

Appending Multiple Lines with sed

cat and echo can only append to the end of a file, while sed can insert at a specific line.

# Add two lines at the beginning of t1.log
sed -i '1i Line1
Line2' t1.log

Practice: Modify nginx Configuration (Line 39)

# Show result without changing the file
sed '39 i listen 81;' /etc/nginx/nginx.conf
# Apply change permanently
sed -i '39 i listen 81;' /etc/nginx/nginx.conf

Practice: Modify sshd_config

# Backup original file
cp /etc/ssh/sshd_config{,.ori}
# Insert multiple lines at the top
sed -i '1i Port 25515
PermitRootLogin no
PerminEmptyPasswords no
UseDNS no
GSSAPIAuthentication no' /etc/ssh/sshd_config

sed Delete Commands

d  Delete the addressed line.
1,4d  Delete lines 1 through 4.
3,$d  Delete from line 3 to the end.
/game/d  Delete lines containing "game".
/game/,$d  Delete from the line containing "game" to the end.
/game/,+1 d  Delete the line containing "game" and the following line.
/^[0-9]{9}/p -n  Print lines that contain a nine‑digit number.

Printing Line Ranges

sed -n '1,3p' file   # Print first three lines
sed -n '2p' file      # Print the second line

sed Grouped Substitution

# Extract the word "welcome" from a sentence
echo 'I am teacher yuchao,welcome my linux course' | \
  sed -r 's/^.*,(.*)m.*/\1/g'

Extract IP Address Example

# Using ifconfig and sed to extract the IPv4 address
ifconfig ens33 | sed -e '2s/^.*inet//' -e '2s/netmask.*//p' -n

sed Query and Print Commands

# Print the second line
sed -n '2p' t1.log
# Print lines containing "http" or "linux"
sed -e '/http/p' -e '/linux/p' t1.log -n

Other sed Commands

w – Write the selected lines to a file.

Example: Write lines containing "computer" to game2.log <code>sed -n '/computer/w game2.log' t1.log</code>

Example: Replace all occurrences of "yuchao" with "老于" and write to yu.log <code>sed -n 's#yuchao#老于#gw yu.log' t1.log</code>

-e – Combine multiple commands. <code>sed -e '1p' -e '2p' -e '4p' t1.log -n</code>

Semicolon can also separate commands on a single line. <code>sed '1p;2p;4p' t1.log -n</code>

Source: https://www.cnblogs.com/btcm409181423/p/18016756#top (© original author)