Master Sed: The Ultimate Guide to Automating Text Processing

sed Editor

sed is a stream editor (Stream Editor) used for line‑by‑line processing of text files. It reads each line into a temporary buffer called the pattern space, applies editing commands, and outputs the result to standard output or a file. By default it does not modify the source file unless the -i option is used.

Workflow

Read : sed reads a line from the input stream (file, pipe, or stdin) into the pattern space.

Execute : It applies the specified commands to the pattern space. Commands are executed sequentially unless an address range limits them.

Display : The processed line is written to stdout or a specified file. The pattern space is then cleared and the cycle repeats until all input is processed.

Improving Execution Efficiency

When handling large files, sed may become slower. Two common optimization methods are:

Method 1 (recommended) : Use the split command to divide a large file into smaller chunks and run sed on each chunk.

# Split a million‑line file into 10 000‑line pieces
split -l 10000 largefile.txt smallfile_
# Split by size (e.g., 400 MB)
split -b 400M test1.txt se

Method 2 : Pipe the file through cat or sed directly, which works for medium‑sized files but may be less effective for very large ones.

cat /etc/passwd | sed ''

Command Syntax and Options

The basic syntax is: sed ' /pattern/command arguments' filename Common ways to specify commands:

sed -e 'operation' file1 [file2]
# Use -e for multiple operations
sed -f scriptfile file1 [file2]
# In‑place editing (use with caution)
sed -i -e 'operation' file [file2]

Frequently used options:

-e or --expression=: specify a sed command (optional when only one command is given).

-f or --file=: read commands from a script file.

-h or --help: display help.

-i : edit files in place.

-n : suppress automatic printing; usually combined with p to print selected lines.

-r (or -E on BSD): enable extended regular expressions.

Addressing (Line Selection)

Addresses determine which lines a command applies to. Types include:

Line number : e.g., sed '2d' file.txt deletes line 2.

Line range : e.g., sed '2,4d' file.txt deletes lines 2‑4.

Pattern match : e.g., sed '/pattern/d' file.txt deletes lines containing pattern.

Combination : e.g., sed '1,3s/foo/bar/' file.txt replaces foo with bar on lines 1‑3.

Core Functions

sed provides four main capabilities: add, delete, change, and print.

Printing (p, =, l commands)

By default all lines are printed. Use -n to suppress default output and p to print selected lines.

# Print all lines (default)
sed '' /etc/fstab
# Print only line 2
sed -n '2p' test1.txt
# Print line numbers and content
sed -n '=;p' test1.txt

Deletion (d command)

Delete lines by number, range, or pattern. Example:

# Delete line 3
sed '3d' file.txt
# Delete lines containing "one"
sed '/one/d' file.txt
# Delete empty lines
sed '/^$/d' file.txt

Replacement (s, c, y commands)

The s command substitutes text. Syntax: address s/old/new/flags.

# Replace first occurrence on each line
sed 's/old/new/' file.txt
# Replace all occurrences
sed 's/old/new/g' file.txt
# Replace only the second occurrence
sed 's/old/new/2' file.txt

Flags: g: global replacement. p: print the line after substitution (used with -n). w file: write the result to file.

Other commands: c: replace the entire line. y: transliterate characters (must have equal length source and target sets).

Case Conversion

Special escape sequences can change case:

# Convert uppercase to lowercase
sed 's/[A-Z]/\l&/g' file.txt
# Capitalize first character of each match
sed 's/[a-z]/\u&/' file.txt
# Convert entire match to uppercase
sed 's/[a-z]/\U&/g' file.txt

Insertion and Appending (a, i, r commands)

Add text after or before a matching line, or read from another file.

# Append after lines containing "three"
sed '/three/a 123' file.txt
# Insert before lines containing "three"
sed '/three/i 123' file.txt
# Read contents of another file after a match
sed '/three/r other.txt' file.txt
# Append at end of file
sed '$a 123' file.txt

Using -f to Read Command Files

The -f option lets you store sed commands in a separate script file and apply them to one or more input files.

# script.txt contains: s/ /_/g
sed -f script.txt input.txt
# Replaces all spaces with underscores

These examples cover the essential features of sed for automating text processing tasks, from basic line selection to complex substitutions and script‑based workflows.