Mastering curl: Fetch, Save, and Manipulate Web Content from the Command Line

Fetching page content

When curl is run without options, it sends a GET request and prints the response to standard output.

curl http://www.codebelief.com

Displaying HTTP headers

Use -I to retrieve only the response headers. curl -I http://www.codebelief.com Example output:

HTTP/1.1 200 OK
Server: nginx/1.10.3
Date: Thu, 11 May 2017 08:24:45 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 24206
Connection: keep-alive
X-Powered-By: Express
Cache-Control: public, max-age=0
ETag: W/"5e8e-Yw5ZdnVVly9/aEnMX7fVXQ"
Vary: Accept-Encoding

Showing headers and body together

Use -i to include headers in the output along with the page body. curl -i http://www.codebelief.com The output starts with the same headers followed by the HTML content.

Saving output to a file

Redirect with > or use the -o / -O options. curl http://www.codebelief.com > index.html Options: -o filename: save to the specified filename. -O: use the remote filename from the URL.

curl -o index.html http://www.codebelief.com

curl -O http://www.codebelief.com/page/2/

Note: -O requires the URL to end with a filename; otherwise use -o to specify one.

Downloading multiple files

Specify several URLs with repeated -O or -o options.

curl -O http://www.codebelief.com/page/2/ -O http://www.codebelief.com/page/3/

curl -o page1.html http://www.codebelief.com/page/1/ -o page2.html http://www.codebelief.com/page/2/

Following redirects

Use -L to follow HTTP 301/302 redirects automatically.

curl -L http://codebelief.com

Custom User-Agent

Set a custom User-Agent string with -A.

curl -A "Mozilla/5.0 (Android; Mobile; rv:35.0) Gecko/35.0 Firefox/35.0" http://www.baidu.com

Custom request headers

Add arbitrary headers using -H, for example Referer or a custom User-Agent.

curl -H "Referer: www.example.com" -H "User-Agent: Custom-User-Agent" http://www.baidu.com

Cookies can also be sent via -H:

curl -H "Cookie: JSESSIONID=D0112A5063D938586B659EF8F939BE24" http://www.example.com

Saving and loading cookies

Use -c to write cookies to a file and -b to read them.

curl -c cookie-example http://www.example.com

curl -b cookie-example http://www.example.com

-b

accepts either a raw cookie string or a filename containing cookies.

Sending POST data

Use -d to provide form data and -X POST to set the request method.

curl -d "userName=tom&passwd=123456" -X POST http://www.example.com/login

Data can also be sent with a GET request by forcing the method or using -G:

curl -d "somedata" -X GET http://www.example.com/api

curl -d "somedata" -G http://www.example.com/api

Reading data from a file

Prefix the filename with @ to read request body from a file.

curl -d "@data.txt" http://www.example.com/login

Login with cookies

Combine -c to store cookies and -d to post credentials.

curl -c cookie-login -d "userName=tom&passwd=123456" http://www.example.com/login

Subsequent requests can reuse the saved cookie: curl -b cookie-login http://www.example.com/login This maintains the logged‑in session across multiple curl invocations.