Mastering curl: Transfer URLs and Retrieve Data on Linux

curl is a versatile command‑line tool that supports many protocols (DICT, FILE, FTP, FTPS, GOPHER, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTSP, SCP, SFTP, SMTP, SMTPS, TELNET, TFTP) for sending data to a server or receiving data from a server.

Command format

curl [options] [URL...]

Access a URL

curl http://www.bilibili.com

Save the retrieved page

curl http://www.bilibili.com >> index.html

Use built‑in -o to save a page

curl -o index.html http://www.bilibili.com

Use built‑in -O to save a file with its original name

curl -O http://www.bilibili.com/main.css

Test HTTP response code

curl -o /dev/null -s -w %{http_code} www.bilibili.com

GET mode example

curl http://www.bilibili.com/islogin.cgi?param1=abcdef&param2=12345

Skip SSL verification

curl -k 'https://www.bilibili.com/islogin.cgi?param1=abcdef&param2=12345'

POST mode

curl -d "param1=abcdef&param2=12345" http://www.bilibili.com/login.cgi

Specify a proxy server and port

curl -x 172.16.71.233:1080 http://www.bilibili.com

Save cookies from the response

curl -c cookie.txt http://www.bilibili.com

Save response headers

curl -D cookied.txt http://www.bilibili.com

Note: the cookie file created with -c differs from the one saved with -D.

Use saved cookies in a request

curl -b cookie.txt http://www.bilibili.com

Imitate a browser user‑agent

curl -A "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.0)" http://www.bilibili.com

Forge a Referer header (anti‑hotlinking)

curl -e "www.bilibili.com" http://live.bilibili.com

This makes the server think the request originated from www.bilibili.com.

Looped download

curl -O http://www.bilibili.com/dodo[1-5].JPG

FTP file download

curl -O -u UserName:Password ftp://www.bilibili.com/banner.JPG

curl -O ftp://UserName:[email protected]/banner.JPG

Further study

man curl