Mastering wget: Essential Commands and Advanced Options for Linux Downloads

Overview

wget is a command‑line tool on Linux that downloads files over HTTP, HTTPS and FTP. It can run in the background, resume interrupted transfers, follow links to recreate remote directory structures, and automatically retry on network failures, making it indispensable for system administrators and developers.

Basic Syntax

The general command format is:

wget [options] URL

Key Features

Supports HTTP, HTTPS and FTP protocols, with optional proxy configuration.

Can operate in background mode ( -b) so the user can log out while the download continues.

Automatic retry and resume capabilities ( -c, -t).

Recursive download mode that follows links and rebuilds the remote site locally, respecting /robots.txt.

Ability to convert remote links to local ones for offline browsing.

Common Options

General control

-V, --version

– display version and exit. -h, --help – show help. -b, --background – run in background. -e, --execute=COMMAND – execute a command from .wgetrc.

Logging

-o, --output-file=FILE

– write log to FILE. -a, --append-output=FILE – append log to FILE. -d, --debug – enable debug output. -q, --quiet – suppress output. -v, --verbose – default verbose mode.

Download control

-t, --tries=NUMBER

– maximum retry attempts (0 = unlimited). -c, --continue – resume partially downloaded files. -O, --output-document=FILE – save to a specific filename. --limit-rate=RATE – throttle download speed. -Q, --quota=NUMBER – limit total download size (effective for recursive downloads).

Directory handling

-nd, --no-directories

– do not create directories. -x, --force-directories – create full directory hierarchy. -nH, --no-host-directories – omit host name directory. -P, --directory-prefix=PREFIX – save files under PREFIX.

HTTP options

--http-user=USER

and --http-passwd=PASS – set HTTP authentication. -E, --html-extension – save HTML files with .html extension. --header=STRING – add custom header. -U, --user-agent=AGENT – specify User‑Agent string.

FTP options

--ftp-user=USER

and --ftp-password=PASS – FTP authentication. --passive-ftp – use passive mode (default). --active-ftp – use active mode.

Recursive download options

-r, --recursive

– enable recursion (use with care). -l, --level=NUMBER – maximum recursion depth. -k, --convert-links – convert links to local files. -m, --mirror – shortcut for -r -N -l inf -nr. -A, --accept=LIST and -R, --reject=LIST – filter by file extensions. -np, --no-parent – do not ascend to parent directories.

Practical Examples

Example 1: Download a single file

wget http://www.linuxidc.com/linuxidc.zip

Example 2: Save with a different name

wget -O wordpress.zip http://www.linuxidc.com/download.aspx?id=1080

Example 3: Limit download speed

wget --limit-rate=300k http://www.linuxidc.com/linuxidc.zip

Example 4: Resume an interrupted download

wget -c http://www.linuxidc.com/linuxidc.zip

Example 5: Run download in background

wget -b http://www.linuxidc.com/linuxidc.zip

Example 6: Spoof User‑Agent

wget --user-agent="Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.16 (KHTML, like Gecko) Chrome/10.0.648.204 Safari/534.16" http://www.linuxidc.com/linuxidc.zip

Example 7: Test a URL without downloading

wget --spider URL

Example 8: Increase retry attempts

wget --tries=40 URL

Example 9: Download multiple URLs from a file

wget -i filelist.txt

Example 10: Mirror an entire website

wget --mirror -p --convert-links -P ./LOCAL URL

Example 11: Exclude certain file types

wget --reject=gif URL

Example 12: Write log to a file

wget -o download.log URL

Example 13: Limit total download size (recursive only)

wget -Q5m -i filelist.txt

Example 14: Download only PDFs recursively

wget -r -A.pdf URL

Example 15: FTP download with authentication

wget --ftp-user=USERNAME --ftp-password=PASSWORD ftp://example.com/file.zip

Compilation from Source (Optional)

To build wget from source:

tar zxvf wget-1.9.1.tar.gz

cd wget-1.9.1

./configure

make

make install