Master wget: 8 Essential Commands for Efficient File Downloading

1. Download files with wget

Use wget followed by a URL to download a file; the file is saved in the current directory with its original name. wget http://www.lxlinux.net To display the file content on standard output instead of saving, add the --output-document - option and pipe the result.

wget http://www.lxlinux.net --output-document - | head -n4

To rename the downloaded file, use -O or --output-document with the desired filename.

wget http://www.lxlinux.net -O newfile.html

2. Resume interrupted downloads

When a download is interrupted, the -c or --continue option resumes it without restarting.

wget -c https://www.lxlinux.net/linux-distro.iso

3. Download multiple files

Use brace expansion in the URL to fetch a series of similarly named files.

wget http://www.lxlinux.net/file_{1..4}.txt

4. Mirror an entire website

Use --mirror (equivalent to --recursive --level=inf --timestamping --no-remove-listing) to download a full site with its directory structure.

Additional options such as --no-cookies --page-requisites --convert-links ensure all pages are complete and links are adjusted for local browsing.

5. Modify HTTP request headers

Use --debug to view the request headers sent by wget. The --header option lets you set custom headers, useful for simulating specific browsers.

wget --debug --header="User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36 Edg/91.0.864.59" http://www.lxlinux.net

wget --debug \
--header="User-Agent: Mozilla/5.0 (iPhone; CPU iPhone OS 13_5_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.1.1 Mobile/15E148 Safari/604.1" http://www.lxlinux.net

6. View response headers

Similarly, --debug displays response headers such as Content-Type, Server, and others.

wget --debug www.lxlinux.net

7. Handle 301 redirects

Use --max-redirect to control how many redirects wget follows; setting it to 0 stops following redirects.

wget --max-redirect 0 http://www.lxlinux.net

8. Expand shortened URLs

Wget can also resolve short links to their final destinations by using --max-redirect 0 on the short URL.

wget --max-redirect 0 "https://bit.ly/2yDyS4T"