Diving into curl as an API developer

If you’ve been a developer for some time now, chances are you’ve heard of ‘curl’. Maybe you’ve stumbled across a reference to it on an API documentation page, then copied and pasted a command and thought nothing more of it.

Many times I find that tools like curl have a lot of hidden powers that only become apparent upon closer inspection. In fact, as we’ll soon discover together, you can use curl to do practically anything you need to accomplish in regards to crafting requests and transferring data between machines.

In this article, I look more closely at what curl is and dive into the details of its abilities to transmit data using a wide variety of different transfer protocols and particularly how it can be used to communicate with APIs.

curl background

curl (cURL) as in ‘Client URL’ has been around since 1997 and has steadily evolved into its present form with the latest version being released just last month at the time of writing.

curl is an open-source project consisting of a library named ‘libcurl’ (written in C) and a command-line program ‘curl’.

curl can be used to transfer data to and from a server using practically every well-known protocol. This includes everything from FTP and HTTP through SMB and SMTP.

I’ve included an extract from the official curl website below which gives a sense of the powerful range of supported protocols and features on offer.

Supports…

DICT, FILE, FTP, FTPS, Gopher, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, MQTT, POP3, POP3S, RTMP, RTMPS, RTSP, SCP, SFTP, SMB, SMBS, SMTP, SMTPS, Telnet and TFTP. curl supports SSL certificates, HTTP POST, HTTP PUT, FTP uploading, HTTP form based upload, proxies, HTTP/2, HTTP/3, cookies, user+password authentication (Basic, Plain, Digest, CRAM-MD5, NTLM, Negotiate and Kerberos), file transfer resume, proxy tunneling and more.

That’s a pretty impressive list!

The curl website also highlights how widely used curl is, as follows.

What’s curl used for?

curl is used in command lines or scripts to transfer data. It is also used in cars, television sets, routers, printers, audio equipment, mobile phones, tablets, settop boxes, media players and is the internet transfer backbone for thousands of software applications affecting billions of humans daily.

That’s pretty cool!

Since the libcurl library is written in C it is well-suited to embedded systems like many of those listed above. However, the curl command-line program which sits on top of libcurl can be utilised to great effect for a variety of other purposes.

As developers, and particularly for web developers, we are usually interested in the more common protocols such as HTTP/HTTPS.

One of the great things about curl is that it uses its own custom HTTP stack, so it is not dependent on a particular browser implementation or on the operating system. This means that you’ll get consistent behaviour across every environment where you use curl.

curl setup

curl is a built-in tool on Linux and Mac.

curl is also built into Windows as of Windows 10 version 1803 which was released in May 2018.

If you are a developer on Windows and use Git for source control, a version of curl can be found in your ‘Git for Windows’ installation folder (%programfiles%\Git\mingw64\bin\curl.exe).

If you don’t already have curl on your system for whatever reason you can get it from the curl download page. Scroll down to the Windows section of the ‘Packages’ table to find the most appropriate version of curl for your system. Download and extract the curl EXE to your machine and add it to your PATH.

curl commands are identical on every platform so you can follow along with the steps which are detailed in the sections below regardless of your operating system.

curl basics

Let’s try out a few basic curl commands using our terminal (e.g. cmd or Windows Terminal).

First up, how can we call an HTTP API endpoint which does not require authentication?

Let’s make use of the JSONPlaceholder API which is a very useful website for testing out API clients.

curl https://jsonplaceholder.typicode.com/todos/1

After issuing the above command you should see a result similar to the following.

{
  "userId": 1,
  "id": 1,
  "title": "delectus aut autem",
  "completed": false
}

Simples!

The usage of curl can be described as follows.

curl [options...] <url>

Note that options can also be specified after the URL, which in many cases can make an overall command more readable.

In the first example, no options were specified. Below is an example of using an option.

curl -i https://jsonplaceholder.typicode.com/todos/1

The -i option instructs curl to include the response headers in addition to the response body. The output following command execution will look similar to the following.

HTTP/1.1 200 OK
Date: Tue, 22 Sep 2020 20:47:47 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 83
Connection: keep-alive
Set-Cookie: __cfduid=d1ef915668cd24fa28ace853be35a79601600807667; expires=Thu, 22-Oct-20 20:47:47 GMT; path=/; domain=.typicode.com; HttpOnly; SameSite=Lax
X-Powered-By: Express
X-Ratelimit-Limit: 1000
X-Ratelimit-Remaining: 999
X-Ratelimit-Reset: 1600662050
Vary: Origin, Accept-Encoding
Access-Control-Allow-Credentials: true
Cache-Control: max-age=43200
Pragma: no-cache
Expires: -1
X-Content-Type-Options: nosniff
Etag: W/"53-hfEnumeNh6YirfjyjaujcOPPT+s"
Via: 1.1 vegur
CF-Cache-Status: HIT
Age: 1665
Accept-Ranges: bytes
cf-request-id: 05592b9f900000075a581f5200000001
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Server: cloudflare
CF-RAY: 5d6ee21288b5075a-LHR

{
"userId": 1,
"id": 1,
"title": "delectus aut autem",
"completed": false
}

The -I option is similar but only includes the response headers i.e. the response body will be excluded. The --head option also has the same effect.

Note that most options have both a ‘dash’ and a ‘dash-dash’ version.

Going further

So what else can we do with curl?

A lot!

Run the following command to retrieve help info.

curl --help

I have included the full output of the command below for reference.

Usage: curl [options...] <url>
     --abstract-unix-socket <path> Connect via abstract Unix domain socket
     --anyauth       Pick any authentication method
 -a, --append        Append to target file when uploading
     --basic         Use HTTP Basic Authentication
     --cacert <CA certificate> CA certificate to verify peer against
     --capath <dir>  CA directory to verify peer against
 -E, --cert <certificate[:password]> Client certificate file and password
     --cert-status   Verify the status of the server certificate
     --cert-type <type> Certificate file type (DER/PEM/ENG)
     --ciphers <list of ciphers> SSL ciphers to use
     --compressed    Request compressed response
 -K, --config <file> Read config from a file
     --connect-timeout <seconds> Maximum time allowed for connection
     --connect-to <HOST1:PORT1:HOST2:PORT2> Connect to host
 -C, --continue-at <offset> Resumed transfer offset
 -b, --cookie <data> Send cookies from string/file
 -c, --cookie-jar <filename> Write cookies to <filename> after operation
     --create-dirs   Create necessary local directory hierarchy
     --crlf          Convert LF to CRLF in upload
     --crlfile <file> Get a CRL list in PEM format from the given file
 -d, --data <data>   HTTP POST data
     --data-ascii <data> HTTP POST ASCII data
     --data-binary <data> HTTP POST binary data
     --data-raw <data> HTTP POST data, '@' allowed
     --data-urlencode <data> HTTP POST data url encoded
     --delegation <LEVEL> GSS-API delegation permission
     --digest        Use HTTP Digest Authentication
 -q, --disable       Disable .curlrc
     --disable-eprt  Inhibit using EPRT or LPRT
     --disable-epsv  Inhibit using EPSV
     --dns-interface <interface> Interface to use for DNS requests
     --dns-ipv4-addr <address> IPv4 address to use for DNS requests
     --dns-ipv6-addr <address> IPv6 address to use for DNS requests
     --dns-servers <addresses> DNS server addrs to use
 -D, --dump-header <filename> Write the received headers to <filename>
     --egd-file <file> EGD socket path for random data
     --engine <name> Crypto engine to use
     --expect100-timeout <seconds> How long to wait for 100-continue
 -f, --fail          Fail silently (no output at all) on HTTP errors
     --fail-early    Fail on first transfer error, do not continue
     --false-start   Enable TLS False Start
 -F, --form <name=content> Specify HTTP multipart POST data
     --form-string <name=string> Specify HTTP multipart POST data
     --ftp-account <data> Account data string
     --ftp-alternative-to-user <command> String to replace USER [name]
     --ftp-create-dirs Create the remote dirs if not present
     --ftp-method <method> Control CWD usage
     --ftp-pasv      Use PASV/EPSV instead of PORT
 -P, --ftp-port <address> Use PORT instead of PASV
     --ftp-pret      Send PRET before PASV
     --ftp-skip-pasv-ip Skip the IP address for PASV
     --ftp-ssl-ccc   Send CCC after authenticating
     --ftp-ssl-ccc-mode <active/passive> Set CCC mode
     --ftp-ssl-control Require SSL/TLS for FTP login, clear for transfer
 -G, --get           Put the post data in the URL and use GET
 -g, --globoff       Disable URL sequences and ranges using {} and []
 -I, --head          Show document info only
 -H, --header <header/@file> Pass custom header(s) to server
 -h, --help          This help text
     --hostpubmd5 <md5> Acceptable MD5 hash of the host public key
 -0, --http1.0       Use HTTP 1.0
     --http1.1       Use HTTP 1.1
     --http2         Use HTTP 2
     --http2-prior-knowledge Use HTTP 2 without HTTP/1.1 Upgrade
     --ignore-content-length Ignore the size of the remote resource
 -i, --include       Include protocol response headers in the output
 -k, --insecure      Allow insecure server connections when using SSL
     --interface <name> Use network INTERFACE (or address)
 -4, --ipv4          Resolve names to IPv4 addresses
 -6, --ipv6          Resolve names to IPv6 addresses
 -j, --junk-session-cookies Ignore session cookies read from file
     --keepalive-time <seconds> Interval time for keepalive probes
     --key <key>     Private key file name
     --key-type <type> Private key file type (DER/PEM/ENG)
     --krb <level>   Enable Kerberos with security <level>
     --libcurl <file> Dump libcurl equivalent code of this command line
     --limit-rate <speed> Limit transfer speed to RATE
 -l, --list-only     List only mode
     --local-port <num/range> Force use of RANGE for local port numbers
 -L, --location      Follow redirects
     --location-trusted Like --location, and send auth to other hosts
     --login-options <options> Server login options
     --mail-auth <address> Originator address of the original email
     --mail-from <address> Mail from this address
     --mail-rcpt <address> Mail from this address
 -M, --manual        Display the full manual
     --max-filesize <bytes> Maximum file size to download
     --max-redirs <num> Maximum number of redirects allowed
 -m, --max-time <time> Maximum time allowed for the transfer
     --metalink      Process given URLs as metalink XML file
     --negotiate     Use HTTP Negotiate (SPNEGO) authentication
 -n, --netrc         Must read .netrc for user name and password
     --netrc-file <filename> Specify FILE for netrc
     --netrc-optional Use either .netrc or URL
 -:, --next          Make next URL use its separate set of options
     --no-alpn       Disable the ALPN TLS extension
 -N, --no-buffer     Disable buffering of the output stream
     --no-keepalive  Disable TCP keepalive on the connection
     --no-npn        Disable the NPN TLS extension
     --no-sessionid  Disable SSL session-ID reusing
     --noproxy <no-proxy-list> List of hosts which do not use proxy
     --ntlm          Use HTTP NTLM authentication
     --ntlm-wb       Use HTTP NTLM authentication with winbind
     --oauth2-bearer <token> OAuth 2 Bearer Token
 -o, --output <file> Write to file instead of stdout
     --pass <phrase> Pass phrase for the private key
     --path-as-is    Do not squash .. sequences in URL path
     --pinnedpubkey <hashes> FILE/HASHES Public key to verify peer against
     --post301       Do not switch to GET after following a 301
     --post302       Do not switch to GET after following a 302
     --post303       Do not switch to GET after following a 303
     --preproxy [protocol://]host[:port] Use this proxy first
 -#, --progress-bar  Display transfer progress as a bar
     --proto <protocols> Enable/disable PROTOCOLS
     --proto-default <protocol> Use PROTOCOL for any URL missing a scheme
     --proto-redir <protocols> Enable/disable PROTOCOLS on redirect
 -x, --proxy [protocol://]host[:port] Use this proxy
     --proxy-anyauth Pick any proxy authentication method
     --proxy-basic   Use Basic authentication on the proxy
     --proxy-cacert <file> CA certificate to verify peer against for proxy
     --proxy-capath <dir> CA directory to verify peer against for proxy
     --proxy-cert <cert[:passwd]> Set client certificate for proxy
     --proxy-cert-type <type> Client certificate type for HTTS proxy
     --proxy-ciphers <list> SSL ciphers to use for proxy
     --proxy-crlfile <file> Set a CRL list for proxy
     --proxy-digest  Use Digest authentication on the proxy
     --proxy-header <header/@file> Pass custom header(s) to proxy
     --proxy-insecure Do HTTPS proxy connections without verifying the proxy
     --proxy-key <key> Private key for HTTPS proxy
     --proxy-key-type <type> Private key file type for proxy
     --proxy-negotiate Use HTTP Negotiate (SPNEGO) authentication on the proxy
     --proxy-ntlm    Use NTLM authentication on the proxy
     --proxy-pass <phrase> Pass phrase for the private key for HTTPS proxy
     --proxy-service-name <name> SPNEGO proxy service name
     --proxy-ssl-allow-beast Allow security flaw for interop for HTTPS proxy
     --proxy-tlsauthtype <type> TLS authentication type for HTTPS proxy
     --proxy-tlspassword <string> TLS password for HTTPS proxy
     --proxy-tlsuser <name> TLS username for HTTPS proxy
     --proxy-tlsv1   Use TLSv1 for HTTPS proxy
 -U, --proxy-user <user:password> Proxy user and password
     --proxy1.0 <host[:port]> Use HTTP/1.0 proxy on given port
 -p, --proxytunnel   Operate through a HTTP proxy tunnel (using CONNECT)
     --pubkey <key>  SSH Public key file name
 -Q, --quote         Send command(s) to server before transfer
     --random-file <file> File for reading random data from
 -r, --range <range> Retrieve only the bytes within RANGE
     --raw           Do HTTP "raw"; no transfer decoding
 -e, --referer <URL> Referrer URL
 -J, --remote-header-name Use the header-provided filename
 -O, --remote-name   Write output to a file named as the remote file
     --remote-name-all Use the remote file name for all URLs
 -R, --remote-time   Set the remote file's time on the local output
 -X, --request <command> Specify request command to use
     --request-target Specify the target for this request
     --resolve <host:port:address> Resolve the host+port to this address
     --retry <num>   Retry request if transient problems occur
     --retry-connrefused Retry on connection refused (use with --retry)
     --retry-delay <seconds> Wait time between retries
     --retry-max-time <seconds> Retry only within this period
     --sasl-ir       Enable initial response in SASL authentication
     --service-name <name> SPNEGO service name
 -S, --show-error    Show error even when -s is used
 -s, --silent        Silent mode
     --socks4 <host[:port]> SOCKS4 proxy on given host + port
     --socks4a <host[:port]> SOCKS4a proxy on given host + port
     --socks5 <host[:port]> SOCKS5 proxy on given host + port
     --socks5-basic  Enable username/password auth for SOCKS5 proxies
     --socks5-gssapi Enable GSS-API auth for SOCKS5 proxies
     --socks5-gssapi-nec Compatibility with NEC SOCKS5 server
     --socks5-gssapi-service <name> SOCKS5 proxy service name for GSS-API
     --socks5-hostname <host[:port]> SOCKS5 proxy, pass host name to proxy
 -Y, --speed-limit <speed> Stop transfers slower than this
 -y, --speed-time <seconds> Trigger 'speed-limit' abort after this time
     --ssl           Try SSL/TLS
     --ssl-allow-beast Allow security flaw to improve interop
     --ssl-no-revoke Disable cert revocation checks (WinSSL)
     --ssl-reqd      Require SSL/TLS
 -2, --sslv2         Use SSLv2
 -3, --sslv3         Use SSLv3
     --stderr        Where to redirect stderr
     --suppress-connect-headers Suppress proxy CONNECT response headers
     --tcp-fastopen  Use TCP Fast Open
     --tcp-nodelay   Use the TCP_NODELAY option
 -t, --telnet-option <opt=val> Set telnet option
     --tftp-blksize <value> Set TFTP BLKSIZE option
     --tftp-no-options Do not send any TFTP options
 -z, --time-cond <time> Transfer based on a time condition
     --tls-max <VERSION> Use TLSv1.0 or greater
     --tlsauthtype <type> TLS authentication type
     --tlspassword   TLS password
     --tlsuser <name> TLS user name
 -1, --tlsv1         Use TLSv1.0 or greater
     --tlsv1.0       Use TLSv1.0
     --tlsv1.1       Use TLSv1.1
     --tlsv1.2       Use TLSv1.2
     --tlsv1.3       Use TLSv1.3
     --tr-encoding   Request compressed transfer encoding
     --trace <file>  Write a debug trace to FILE
     --trace-ascii <file> Like --trace, but without hex output
     --trace-time    Add time stamps to trace/verbose output
     --unix-socket <path> Connect through this Unix domain socket
 -T, --upload-file <file> Transfer local FILE to destination
     --url <url>     URL to work with
 -B, --use-ascii     Use ASCII/text transfer
 -u, --user <user:password> Server user and password
 -A, --user-agent <name> Send User-Agent <name> to server
 -v, --verbose       Make the operation more talkative
 -V, --version       Show version number and quit
 -w, --write-out <format> Use output FORMAT after completion
     --xattr         Store metadata in extended file attributes

I’m sure you’ll agree that based on the above output, it is clear that curl offers a phenomenal and perhaps intimidating number of options.

These options will allow you to accomplish nearly any task you could imagine when it comes to client to server communication.

Let’s go ahead and explore some more things that we can do with curl.

Downloading files

Working with files is very straightforward.

Try running the following command.

curl https://jsonplaceholder.typicode.com/todos/1 -o %userprofile%\Desktop\curl-output.txt

The -o option allows you to specify the location of a file to which the output of the command should be written to. In the above example, a file named ‘curl-output.txt’ will be created in the Desktop directory of the current user.

This also works for downloading files generally.

curl https://via.placeholder.com/600/92c952 -o %userprofile%\Desktop\photo.png

It’s exactly the same syntax as the previous example. In the above example, a PNG photo is saved to the Desktop directory of the current user.

Posting data

How about posting data to a HTTP API?

curl -X POST https://jsonplaceholder.typicode.com/todos -d "{\"userId\": 11,\"id\": 201,\"title\": \"Leave out the bin\",\"completed\": false}"

The -X option is used to specify the request method i.e. POST.

The -d option is used to specify the data to post.

In many cases, headers will also need to be included in the request and these can be specified by using a -H option for each required header.

curl -X POST -H "Content-Type: application/json" -H "Accept: application/json" https://jsonplaceholder.typicode.com/todos -d "{\"userId\": 11,\"id\": 201,\"title\": \"Leave out the bin\",\"completed\": false}"

Instead of JSON, we could post key-value pairs for form data in a similar manner and update the Content-Type as required e.g. title=Hello&body=Hello World

Updates & deletes

Updates are similar to posts, we just need to specify PUT as the request method.

curl -X PUT https://jsonplaceholder.typicode.com/todos/201 -d "{\"userId\": 11,\"id\": 201,\"title\": \"Leave out the bin again\",\"completed\": false}"

Deletes are just as easy to accomplish.

curl -X DELETE https://jsonplaceholder.typicode.com/todos/201

Authentication

If authentication details are required for an endpoint there are a number of different options for handling this.

For example, to access an endpoint that requires Basic authentication we can use the -u option as follows.

curl -u user:pw https://jsonplaceholder.typicode.com/todos/1

Many APIs are protected by authentication protocols that rely on HTTP headers.

As per the Posting data sub-section above, we can specify headers using the -H option.

curl -H "Authorization: Bearer {token}" https://jsonplaceholder.typicode.com/todos/1

‘Bearer’ token authentication is used in the above command.

For other authentication types, it’s just a matter of swapping out the header to suit the particular style of authentication e.g. some APIs use a custom ‘Api-Key’ style header.

Troubleshooting issues

If you need to troubleshoot a request issue you can add the -v option to specify verbose output. This will display logs regarding the command you have issued and will also include the response headers.

curl -v https://jsonplaceholder.typicode.com/todos/1

I have included a partial extract of the command output below for reference.

* Trying 172.64.197.36...
* TCP_NODELAY set
* Connected to jsonplaceholder.typicode.com (172.64.197.36) port 443 (#0)
* schannel: SSL/TLS connection with jsonplaceholder.typicode.com port 443 (step 1/3)
* schannel: checking server certificate revocation
* schannel: sending initial handshake data: sending 193 bytes...
* schannel: sent initial handshake data: sent 193 bytes
* schannel: SSL/TLS connection with jsonplaceholder.typicode.com port 443 (step 2/3)
* schannel: failed to receive handshake, need more data
* schannel: SSL/TLS connection with jsonplaceholder.typicode.com port 443 (step 2/3)
* schannel: encrypted data got 2420
* schannel: encrypted data buffer: offset 2420 length 4096
* schannel: sending next handshake data: sending 93 bytes...
* schannel: SSL/TLS connection with jsonplaceholder.typicode.com port 443 (step 2/3)
* schannel: encrypted data got 258
* schannel: encrypted data buffer: offset 258 length 4096
* schannel: SSL/TLS handshake complete
* schannel: SSL/TLS connection with jsonplaceholder.typicode.com port 443 (step 3/3)
* schannel: stored credential handle in session cache
> GET /todos/1 HTTP/1.1
...

This detailed information can be very useful for tracking down communication problems e.g. firewall or SSL certificate issues.

Other tools

Before finishing up I wanted to mention a couple of additional tools related to curl.

Postman

If you’ve ever worked with REST APIs you’ve almost certainly heard of Postman.

Postman is a great GUI tool that can assist you with building and testing APIs. It allows you to create and save requests and facilitates collaboration with other developers.

Postman can generate curl commands.

After you have defined a request in Postman, press the ‘Code’ link. This will launch the ‘Generate Code Snippets’ dialog which defaults to cURL.

You can copy and paste the generated command into your terminal to try it out.

ReqBin

ReqBin is an online API testing tool that lets you issue requests from your web browser.

ReqBin has a dedicated curl section where you can try out curl commands without needing to have curl installed on your local machine.

Most of the commands which I have demonstrated in the previous sections will work just fine on ReqBin.

Summary

curl is a great addition to any developer’s toolkit and it allows us to accomplish pretty much any file transfer task that we need to automate with its plethora of available options.

curl gives API developers a lot of control over how operations and requests are crafted and it is also a very powerful automation tool.

Because curl is a command-line tool it integrates nicely with automated processes. You can customise the output returned by curl so that the data can be fed into other commands.

Having tried out the most common commands of interest to API developers, I trust that you now feel empowered to use curl as and when needed.


I hope you enjoyed this post! Comments are always welcome and I respond to all questions.

If you like my content and it helped you out, please check out the button below 🙂

Comments