Essential Shell Commands and Scripting Techniques You Must Know

Variable Definition and Usage

Variable definition and usage

Spaces are optional in shell except in assignments and parameters. Positional parameters are accessed as $1 … $n without spaces. Use braces to avoid ambiguity.

var=1
echo $var
# Recommended: expand with braces
echo "this is the ${var}nd"
# Without braces $varnd is interpreted as a different variable
echo "this is thi $varnd"
# Single quotes prevent expansion
echo 'this is the ${var}nd'
readonly var
var=2   # error: var is read‑only
var1=3
unset var1   # delete variable

Double quotes allow strings with spaces to be treated as a single word.

#!/bin/bash
var="/home/My Picture"
if [ "/home/My Picture" == "$var" ]; then
  echo "find the path"
else
  echo "no path"
fi

Arithmetic Operations

Arithmetic operations

var=0
((var+=1))   # var==1
((var++))    # var==2
((var=var*var))   # var==4
let var=var/3   # var==1
let 'var= var / 3'   # spaces require quotes
echo $((var+=2))   # prints 2 (C‑style expression)

var=1
var=`expr "$var" + 1`   # result 2, spaces required
echo `expr "$var" + 1`   # prints 3

Backticks execute a command and assign its output to a variable.

String Operations

String concatenation

your_name="runoob"
greeting_1="hello, ${your_name} !"
greeting_2='hello, ${your_name} !'
echo $greeting_1 $greeting_2   # hello, runoob ! hello, ${your_name} !

String length

string="abcd"
echo ${#string}   # outputs 4

Substring extraction (index starts at 0)

string="runoob is a great site"
echo ${string:1:4}   # outputs unoo

Arrays (one‑dimensional, unlimited size)

Array definition

array_name=(value0 value1 value2)
# multiline definition
array_name=(
value0
value1
value2
)
array_name[0]=value0
array_name[1]=value1
array_name[n]=valuen   # indices need not be consecutive

Read array

echo ${array_name[0]}   # value0
echo ${array_name[@]}   # all elements

Array length

length=${#array_name[@]}   # number of elements
length=${#array_name[*]}
lengthn=${#array_name[n]}   # length of element n

Quoting

#!/bin/bash
echo $SHELL   # => /bin/bash
echo "$SHELL"   # => /bin/bash
echo '$SHELL'   # => $SHELL
echo \$SHELL   # => $SHELL

your_name='runoob'
str="Hello, I know you are \"$your_name\"! 
"   # without braces
echo -e $str   # 
 printed literally
str="Hello, I know you are \"${your_name}\"! 
"   # with braces
echo -e $str

Single‑quote strings output characters literally; variables are not expanded and a single quote cannot appear alone inside single quotes. Double quotes allow variable expansion and interpretation of escape characters.

Test, Bracket, Arithmetic and Let Differences

test [ ] [[ ]] (( )) let differences

test and [ ] perform file, string and integer tests. [[ ]] is more powerful; it supports && and || inside the expression. Spaces are required around each token.

When multiple expressions need to be combined in [ ], use boolean operators. let and (( )) are similar for arithmetic; comparison operators (<, >, ==) can be used without the $ prefix.

Boolean Operators

!   # NOT
-o   # OR
-a   # AND

File Comparison Operators

-e "${filename}"   # file exists
-d "${filename}"   # is a directory
-f "${filename}"   # is a regular file
-L "${filename}"   # is a symbolic link
-r "${filename}"   # readable
-w "${filename}"   # writable
-x "/bin/ls"       # executable
"${filename1}" -nt "${filename2}"   # newer than
"${filename1}" -ot "${filename2}"   # older than

String Comparison Operators

-z "${myvar}"   # true if empty
-n "${myvar}"   # true if non‑empty
"${a}" == "${b}"   # equality
"${a}" != "${b}"   # inequality

Arithmetic Comparison Operators (used inside [[ ]] )

3 -eq "$mynum"   # equal
3 -ne "$mynum"   # not equal
3 -lt "$mynum"   # less than
3 -le "$mynum"   # less or equal
3 -gt "$mynum"   # greater than
3 -ge "$mynum"   # greater or equal

Control Flow

if statement

#!/bin/bash
if [ "${SHELL}" == "/bin/bash" ]; then
  echo "your login shell is bash"
else
  echo "your login shell is not bash but ${SHELL}"
fi

&& and || operators

Simplified if using logical AND:

[ -f "/etc/shadow" ] && echo "This computer uses shadow passwords"

Logical OR runs the second command only if the first fails. Braces can group commands.

#!/bin/bash
mailfolder=/var/spool/mail/james
[ -r "${mailfolder}" ] || { echo "Cannot read ${mailfolder}"; exit 1; }
echo "${mailfolder} has mail from:"
grep "^From" ${mailfolder}

case statement

#!/bin/bash
ftype=`file $1`
case "${ftype}" in
  "$1: Zip archive"*)
    unzip "$1";;
  "$1: gzip compressed"*)
    gunzip "$1";;
  "$1: bzip2 compressed"*)
    bunzip2 "$1";;
  *)
    echo "File $1 cannot be uncompressed with smartzip";;
esac

;;

acts like break. The closing parenthesis ) ends the pattern list.

select loop

#!/bin/bash
pocket=()
select var in 跳跳糖 糖 很多糖 企鹅糖; do
  echo "除了 $var 还要什么吗？"
  if ((RANDOM%4 == 0)); then
    echo "呀！时间不够了，快上车！"
    break
  fi
  pocket+=("$var")
 done
 echo "你最后说的那个 $var 弄丢了……"
 IFS='、'
 echo "现在口袋里只有：${pocket[*]}。"
 IFS=$' \t
'

#!/bin/bash
echo "What is your favourite OS?"
select var in "Linux" "Gnu Hurd" "Free BSD" "Other"; do
  break
 done
echo "You have selected ${var}"

select

repeatedly prompts until break is called.

while / for loops

#!/bin/bash
var=10
while ((var>0)); do
  ((var--))
  echo $var
done

#!/bin/bash
for var in "Linux" "GNU Hurd" "Free BSD" "Other"; do
  echo $var
done

#!/bin/bash
# List a summary of RPM packages
# USAGE: showrpm rpmfile1 rpmfile2 ...
# EXAMPLE: showrpm /cdrom/RedHat/RPMS/*.rpm
for rpmpackage in "$@"; do
  if [ -r "$rpmpackage" ]; then
    echo "=============== $rpmpackage ==============="
    rpm -qi -p $rpmpackage
  else
    echo "ERROR: cannot read file $rpmpackage"
  fi
done

$@

behaves the same with or without double quotes; $* becomes a single string when quoted.

Here Document

A here‑document starts with <<WORD and ends with the same WORD on a line by itself.

cat<<HELP
Description text
HELP

Example with argument count check:

#!/bin/bash
if [ $# -lt 3 ]; then
  cat <<HELP
ren -- renames a number of files using sed regular expressions
USAGE: ren 'regexp' 'replacement' files...
EXAMPLE: rename all *.HTM files to *.html:
  ren 'HTM$' 'html' *.HTM
HELP
  exit 0
fi
OLD="$1"
NEW="$2"
shift
shift
for file in "$@"; do
  if [ -f "$file" ]; then
    newfile=`echo "$file" | sed "s/${OLD}/${NEW}/g"`
    if [ -f "$newfile" ]; then
      echo "ERROR: $newfile exists already"
    else
      echo "renaming $file to $newfile ..."
      mv "$file" "$newfile"
    fi
  fi
done

Functions

#!/bin/bash
help(){
cat <<HELP
xtitlebar -- change the name of an xterm, gnome-terminal or kde konsole
USAGE: xtitlebar [-h] "string_for_titlebar"
OPTIONS: -h help text
EXAMPLE: xtitlebar "cvs"
HELP
  exit 0
}
if [[ $1 == '' || $1 == '-h' ]]; then
  help
fi
echo -e "\033]0;$1\007"

Script Practice – Command‑Line Argument Parsing

#!/bin/bash
help(){
  cat <<HELP
This is a generic command line parser demo.
USAGE EXAMPLE: cmdparser -l hello -f -- -somefile1 somefile2
HELP
  exit 0
}
while [ -n "$1" ]; do
  case "$1" in
    -h) help; shift 1;;
    -f) opt_f=1; shift 1;;
    -l) opt_l=$2; shift 2;;
    --) shift; break;;
    -*) echo "error: no such option $1. -h for help"; exit 1;;
    *) break;;
  esac
done

echo "opt_f is $opt_f"
echo "opt_l is $opt_l"
echo "first arg is $1"
echo "2nd arg is $2"

The while loop reads $1. When -l matches, opt_l is set and shift 2 removes both the option and its argument. The loop continues until -- is encountered, after which remaining arguments are processed.

Script Debugging

sh -x strangescript   # trace execution with variable values
sh -n your_script     # syntax check only, no execution

Common Text‑Processing Methods

sort – ordering -f ignore case, -b ignore leading blanks, -M month sort, -n numeric, -r reverse, -u uniq, -t delimiter (default tab), -k field

cat /etc/passwd | sort -t ':' -k 3nr   # sort by 3rd field numerically, reverse

uniq – remove duplicates (needs sorted input) -i ignore case, -c count, -u unique only, -d duplicate only

# cat testfile
hello
world
friend
hello
world
hello

sort testfile | uniq -c
# 1 friend
# 3 hello
# 2 world

sort testfile | uniq -d -c   # show only duplicates with counts
# 3 hello
# 2 world

cut – field extraction -d delimiter, -f fields, -c character range

echo $PATH | cut -d ':' -f 3,5   # /sbin:/usr/local/bin
echo $PATH | cut -d ':' -f 3-5   # /sbin:/usr/sbin:/usr/local/bin
echo $PATH | cut -c 1-4   # /bin
echo $PATH | cut -c 1,4   # /n

wc – word count -l lines, -w words, -m characters, -c bytes

wc -l /etc/passwd   # 40 lines (40 accounts)
wc -w /etc/passwd   # 45 words
wc -m /etc/passwd   # 1719 characters

sed – Stream Editor

Basic commands: d delete, a append after current line, i insert before current line, p print (usually with -n), s substitute (supports regex), g global replace.

-n silent mode (only explicit output is printed)

-i edit file in place

# delete lines 3 to end
nl /etc/passwd | sed '3,$d'
# insert "itcast" after line 2
nl /etc/passwd | sed '2a itcast'
# insert before line 2
nl /etc/passwd | sed '2i itcast'
# print lines 5‑7 only
nl /etc/passwd | sed -n '5,7p'
# extract IP of eth0
ifconfig eth0 | grep 'inet addr' | sed 's/^.*addr://g' | sed 's/Bcast.*$//g'
# replace first occurrence of "test" with "mytest"
sed 's/test/mytest/' example
# replace all occurrences
sed 's/test/mytest/g' example
# print only lines where substitution happened
sed -n 's/^test/mytest/p' example
# edit file in place and add line after line 2
sed -i '3a itcast' passwd.bak

awk – Field‑Oriented Text Processing

Default field separator is space. $0 is the whole line, $1 the first field, $n the nth field.

# last -n 5 shows last five login entries
last -n 5 | awk '{print $1}'   # print first column (user)
# print username and shell from /etc/passwd (colon delimiter)
cat /etc/passwd | awk -F ':' '{print $1"\t"$7}'
# show shell for root entries only
awk -F: '/root/{print $7}' /etc/passwd

grep – Pattern Search

Common options: -a do not ignore binary, -d directory handling, -E extended regex, -i ignore case, -n show line numbers, -s suppress errors, -v invert match.

grep root passwd               # match "root"
grep ^root passwd              # lines starting with root
grep root$ passwd              # lines ending with root
grep -i root passwd            # case‑insensitive
grep -E "root|ROOT" passwd    # extended regex

echo

echo -e "abc
"   # interpret 
 as newline
echo "abc
"      # prints literal 
 then newline
echo -n "abc
"   # no trailing newline
echo -ne "abc
"  # no newline, but 
 interpreted

printf – Formatted Output

printf "%‑10s %‑8s %‑4s
" 姓名 性别 体重kg
printf "%‑10s %‑8s %‑4.2f
" 郭靖 男 66.1234
printf "%‑10s %‑8s %‑4.2f
" 杨过 男 48.6543
printf "%‑10s %‑8s %‑4.2f
" 郭芙 女 47.9876
# output:
# 姓名     性别   体重kg
# 郭靖     男      66.12
# 杨过     男      48.65
# 郭芙     女      47.99

File Attribute Changes

ln – create links Soft link: ln -s source linkname Hard link: ln source linkname (increases link count)

chgrp [-R] group file

chown [-R] owner file

chown [-R] owner:group file

chmod [-R] xyz file

– numeric mode (x for owner, y for group, z for others) chmod [u|g|o|a][+|-|=][r|w|x] file – symbolic mode

# make script executable
chmod 777 while
# remove executable bits
chmod a-x while

File Management

pwd [-P]

– show physical path (no symlinks) mkdir [-mp] dir – -m set mode, -p create parent directories rmdir [-p] dir – remove empty directory, -p also removes empty parents cp source dest – options: -i interactive, -p preserve attributes, -r recursive, -s symbolic link, -l hard link rm [-fir] file/dir – -f force, -i interactive, -r recursive (dangerous) mv [-fiu] src dest – -f force, -i interactive, -u update only if source newer

File Content Viewing

cat

– display from first line tac – display from last line nl – number lines more – page by page less – scroll backward as well as forward head – show beginning lines tail – show ending lines

User and Group Management

useradd options: -c comment, -d home, -m create home, -g primary group, -G supplementary groups, -s login shell, -u UID (with -o allow duplicate UID)

useradd -d /home/sam -m sam   # create user sam with home /home/sam
useradd -s /bin/bash -g group -G adm,root gem   # create user gem

userdel -r also removes home directory

userdel -r sam   # delete user sam and its home

usermod – change shell, home, primary group, etc.

usermod -s /bin/ksh -d /home/z -g developer sam

passwd options: -l lock, -u unlock, -d delete password, -f force change on next login

passwd               # change current user's password (needs old password)
passwd sam           # root can change any user's password without old password
passwd -d sam        # delete password for sam (no password required)

groupadd options: -g GID, -o allow duplicate GID

groupadd group1
groupadd -g 101 group2

groupdel

groupdel group1

groupmod options: -g newGID, -o allow duplicate,

-n newname

groupmod -g 10000 -n group3 group2   # change GID and name

Account‑Related Files

/etc/passwd

– format:

username:password:UID:GID:comment:home:shell

sam:x:200:50:Sam san:/home/sam:/bin/sh

/etc/shadow

– generated by pwconv; format:

login:encrypted:lastchg:min:max:warn:inactive:expire:flags

sam:EkdiSECLWPdSa:9740:0:0::::

/etc/group

– format:

groupname:password:GID:user_list

adm::4:root,adm

Disk Management

df – show filesystem usage -a all filesystems, -k KiB, -m MiB, -h human readable, -H decimal units, -T show type, -i show inode usage

df -ha

du – estimate file/directory space -a all files, -h human readable, -s summary only, -S exclude sub‑directory totals, -k KiB, -m MiB

du -hs   # total size of current directory
du -h    # size of directory and sub‑dirs
du -ha   # size of all files and dirs

fdisk – interactive partitioning tool (example commands shown in help output)

mkfs – create filesystem -t specify type (ext3, ext2, vfat, etc.)

mkfs -t ext3 /dev/sda6

fsck – filesystem check/repair -t filesystem, -A all filesystems in /etc/fstab, -C show progress, -d debug, -p automatic repair, -r ask, -y assume yes, -s sequential, -V verbose

fsck -A -y   # check all filesystems and answer yes to fixes

Reference material:

https://www.jianshu.com/p/a29f32e35dc5

https://blog.csdn.net/su_use/article/details/80742686

https://wiki.ubuntu.org.cn/Shell%E7%BC%96%E7%A8%8B%E5%9F%BA%E7%A1%80

https://www.cnblogs.com/lit10050528/p/4915001.html