Mastering sed: Variable Substitution, Quoting, and Advanced Tricks

1. Problems Using Variables and Variable Substitution in sed

When using sed in scripts you often need to reference shell variables or perform variable substitution inside the sed command line. The difficulty usually lies in quoting; the issue is not with sed itself but with shell quoting rules. Understanding how to solve quoting problems in sed helps you grasp shell quoting and can be applied to other tools such as awk or mysql.

Example: output the last 5 lines of a.txt.

total=`wc -l <a.txt`
sed -n '$((total-4)),$p' a.txt

This fails because $ is special in sed address expressions and the arithmetic expansion $(()) also contains $, causing sed to try to interpret it.

Shell quoting rules:

Single quotes : everything is taken literally; you cannot embed another single quote, even with a backslash.

Double quotes : most characters are literal except \, $, and backticks (and ! when history expansion is enabled).

No quotes : similar to double quotes but brace and tilde expansion occur.

The key point of quoting is that it decides which words the shell parses and which are passed unchanged to commands like sed. Therefore characters that have special meaning to both the shell and sed (e.g., $, !, {}) must be enclosed in single quotes to let sed see them.

Correct ways to print the last 5 lines:

sed -n `$((total-4)),$p' a.txt
sed -n "$(($total-4))",'$p' a.txt
sed -n "$(($total-4)),\$p" a.txt

More examples of variable substitution in sed:

str="abc"
sed -n /^$str/,$p a.txt
sed -n '/^'$str'/,$p' a.txt
sed -n "/^$str/,$p" a.txt

Replacing the password field in /etc/shadow demonstrates the need to change the delimiter because the data contains / and $:

old_pass="$(tail -n 1 /etc/shadow | cut -d':' -f2)"
new_pass='$1$123456$wOSEtcyiP2N/IfIl15W6Z0'
sed -n '$'s%$old_pass%$new_pass%p /etc/shadow

2. Backreference Failure Issue

When an alternation | contains a group that does not participate in the match, backreferences in that branch become invalid. For example, (a)\1u|b\1 only matches aau because the second branch b\1 has no captured group.

3. "-i" Option File Saving Issue

sed -i

creates a temporary file and renames it, so it bypasses file read‑only flags. Whether the rename succeeds depends on the directory's write permission; a read‑only directory prevents -i from saving.

4. Greedy Matching Issue

Basic and extended regular expressions are greedy: they match the longest possible string. To achieve lazy matching you need constructs like .*? (available in PCRE) or use negated character classes, e.g., a[^b]*b. This approach is slower because the engine first matches greedily and then backtracks.

5. Interaction Between "a" and "N" Commands

The a command queues text to be appended after the current output stream. When combined with N, which reads the next line into the pattern space, the queued text may appear before the matched line because N performs an automatic print of the (empty) pattern space before reading.

6. Exclamation Mark Negation in sed

An exclamation mark can appear after an address expression (negating the address) or before a command (negating the command). The two forms have different effects on which lines are processed.

7. sed Hanging and 100% CPU Issue

When processing UTF‑8 files, a mismatch between the locale and file encoding can cause sed to consume 100% CPU. Setting LC_COLLATE and LC_CTYPE (or LANG / LC_ALL) to C resolves the problem.