Mastering cURL: Essential Commands for Downloading, Uploading, and Automation

cURL is a versatile command-line utility for transferring data using over 20 supported protocols, making it ideal for interacting with websites, APIs, and servers. It can download or upload files, emails, or web pages, and is often used within larger scripts for further processing.

Basic File Download

To retrieve a web page or file, use the default HTTP protocol: $ curl http://www.google.com To save the output to a file, add the --output option:

$ curl www.likegeeks.com --output likegeeks.html

Following Redirects

cURL does not follow redirects by default. Use the -L switch to enable this behavior:

$ curl -L www.likegeeks.com

Resuming Interrupted Downloads

Interrupt a transfer with Ctrl+C and resume it using the -C - option:

$ curl -C - example.com/some-file.zip --output MyFile.zip

Specifying Timeouts

Set a maximum execution time with -m (seconds): $ curl -m 60 example.com Set a connection timeout with --connect-timeout:

$ curl --connect-timeout 60 example.com

Authentication with Username and Password

Use -u for basic authentication, e.g., FTP:

$ curl -u username:password ftp://example.com

Using a Proxy

Define an HTTP proxy with -x:

$ curl -x 192.168.1.1:8080 http://example.com

Chunked Download of Large Files

Download specific byte ranges with --range and later combine parts using cat:

$ curl --range 0-99999999 http://releases.ubuntu.com/18.04/ubuntu-18.04.3-desktop-amd64.iso ubuntu-part1
$ curl --range 100000000-199999999 http://releases.ubuntu.com/18.04/ubuntu-18.04.3-desktop-amd64.iso ubuntu-part2
$ cat ubuntu-part? > ubuntu-18.04.3-desktop-amd64.iso

Client Certificate Authentication

Specify a certificate file with --cert:

$ curl --cert path/to/cert.crt:password ftp://example.com

Silent Mode

Suppress progress and error messages with -s. Combine with -O or --output to save files silently:

$ curl -s -O http://example.com
$ curl -s http://example.com --output index.html

Fetching Headers and Titles

Retrieve only the HTTP headers using -I: $ curl -I example.com Follow redirects while fetching headers with -I -L: $ curl -I -L example.com Send custom headers with multiple -H options:

$ curl -H 'Connection: keep-alive' -H 'Accept-Charset: utf-8' http://example.com

Uploading Data and Files

POST form data using -d:

$ curl -d 'name=geek&location=usa' http://example.com

Upload a file with -d @filename or use -T for FTP uploads:

$ curl -d @filename http://example.com
$ curl -T myfile.txt ftp://example.com/some/directory/

Sending Emails via SMTP

Send an email by specifying the SMTP server, sender, recipient, and email file:

$ curl smtp://mail.example.com --mail-from [email protected] --mail-rcpt [email protected] --upload-file email.txt

Reading Emails via IMAP/POP3

List mailboxes with IMAP authentication:

$ curl -u username:password imap://mail.example.com

Fetch a specific message using -X 'UID FETCH 1234':

$ curl -u username:password imap://mail.example.com -X 'UID FETCH 1234'

cURL vs. wget

While both tools retrieve data, wget excels at recursive site downloads and directory traversal, whereas cURL offers broader protocol support and fine-grained control for uploads and custom requests.