Wrapper around LibCURL to make it more Julia like.

This module reexports LibCURL so everything available in LibCURL will be available when this module is used.

See https://curl.se/libcurl/c/libcurl-tutorial.html for a tutorial on using libcurl in C. The Julia interface should be similar.

Examples

GET a URL and read the response from the internal buffer

using CurlHTTP

curl = CurlEasy(
    url="https://postman-echo.com/get?foo=bar",
    method=CurlHTTP.GET,
    verbose=true
)

res, http_status, errormessage = curl_execute(curl)


# curl.userdata[:databuffer] is a Vector{UInt8} containing the bytes of the response
responseBody = String(curl.userdata[:databuffer])

# curl.userdata[:responseHeaders] is a Vector{String} containing the response headers
responseHeaders = curl.userdata[:responseHeaders]

POST to a URL and read the response with your own callback

using CurlHTTP

curl = CurlEasy(
    url="https://postman-echo.com/post",
    method=CurlHTTP.POST,
    verbose=true
)

requestBody = "{"testName":"test_writeCB"}"
headers = ["Content-Type: application/json"]

databuffer = UInt8[]

res, http_status, errormessage = curl_execute(curl, requestBody, headers) do d
    if isa(d, Array{UInt8})
        append!(databuffer, d)
    end
end

responseBody = String(databuffer)

Multiple concurrent requests using CurlMulti

using CurlHTTP

curl = CurlMulti()

for i in 1:3
    local easy = CurlEasy(
        url="https://postman-echo.com/post?val=$i",
        method=CurlHTTP.POST,
        verbose=true,
    )

    requestBody = "{"testName":"test_multi_writeCB","value":$i}"
    headers     = ["Content-Type: application/json", "X-App-Value: $(i*5)"]

    CurlHTTP.curl_setup_request_response(
        easy,
        requestBody,
        headers
    )

    curl_multi_add_handle(curl, easy)
end

res = curl_execute(curl)

responses = [p.userdata for p in curl.pool]  # userdata contains response data, status code and error message

Default user agent to use if not otherwise specified. This allows an application to set the user agent string at init time rather than at constructor time.

Use CurlHTTP.setDefaultUserAgent() to set it. Set it to nothing to unset it.

Internal markers for the data channel

Wrapper around a curl_easy handle. This is what we get from calling curl_easy_init.

Most curl_easy_* functions will work on a CurlEasy object without any other changes.

Summary

struct CurlEasy <: CurlHandle

Fields

handle::Ptr : A C pointer to the curl_easy handle

headers::Ptr : A C pointer to the list of headers passed on to curl. We hold onto this to make sure we can free allocated memory when required.

uuid::UUID : A unique identifier for this curl handle. Used internally to identify a handle within a pool.

userdata::Dict : A dictionary of user specified data. You may add anything you want to this and it will be passed along with the curl handle to all functions. This dictionary will also be populated by several convenience functions to add the :http_status, :errormessage, and response header (:databuffer). All data added by internal code will use Symbol keys.

Constructors

CurlEasy(curl::Ptr) : Create a CurlEasy wrapper around an existing LibCURL curl handle.

CurlEasy(; url::String, method::HTTPMethod, verbose::Bool, certpath::String, keypath::String, cacertpath::String, useragent::String|Nothing) : Create a new curl object with default settings and wrap it. The default settings are:

• FOLLOWLOCATION
• SSL_VERIFYPEER
• SSL_VERIFYHOST
• SSL_VERSION (highest possible up to TLS 1.3)
• HTTP_VERSION (H2 over TLS or HTTP/1.1)
• TCP_FASTOPEN disabled
• TCP_KEEPALIVE
• ACCEPT_ENCODING best supported
• TRANSFER_ENCODING
• DNSCACHETIMEOUT disabled

Additionally the following options are set based on passed in parameters:

• POST if method is POST
• HTTPGET if method is GET
• NOBODY if method is HEAD
• CUSTOMREQUEST if method is HEAD, DELETE, or OPTIONS
• VERBOSE if verbose is true
• SSLCERT if certpath is set
• SSLKEY if certpath and keypath are set
• CAINFO defaults to LibCURL.cacert but can be overridden with cacertpath
• URL if url is set
• USERAGENT if useragent is set to something other than nothing.

Abstract type representing all types of Curl Handles. Currently CurlEasy and CurlMulti.

Wrapper around a curl_multi handle. This is what we get from calling curl_multi_init

Summary

struct CurlMulti <: CurlHandle

Fields

handle::Ptr : A C pointer to the curl_multi handle

pool::CurlEasy[] : An array of CurlEasy handles that are added to this CurlMulti handle. These can be added via the constructor or via a call to curl_multi_add_handle, and may be removed via a call to curl_multi_remove_handle.

Constructors

CurlMulti() : Default constructor that calls curl_multi_init and sets up an empty pool

CurlMulti(::CurlEasy[]) : Constructor that accepts a Vector of CurlEasy objects, creates a curl_multi handle, and adds the easy handles to it.

HTTP Methods recognized by CurlHTTP. Current values are:

• GET: Make a GET request
• POST: Upload data using POST
• HEAD: Make a HEAD request and specify no response body
• DELETE: Make a DELETE request
• PUT: Currently not supported
• OPTIONS: Make an OPTIONS request

Add a vector of headers to the curl object

Run common housekeeping tasks required by a curl callback function.

This function should be called from a curl WRITE or HEADER callback function. It does the following:

1. Calculate the number of bytes read
2. Copy bytes into a Vector{UInt8}
3. Convert any non-null userdata parameter to a julia type

It then returns a tuple of these three values.

Cleanup the CurlHandle automatically determining what needs to be done for curl_easy vs curl_multi handles. In general, this will be called automatically when the CurlHandle gets garbage collected.

[Internal] curl debug callback to log informational text, header data, and SSL data transferred over the network. This will only run if curl is configured in verbose mode.

curl_error_to_string(::Vector{UInt8}) → String

Convert curl's error message stored as a NULL terminated sequence of bytes into a Julia String

curl_execute(data_handler::Function, ::CurlEasy, ::String, ::Vector{String}; url::String) → (CURLCode, Int64, String)
curl_execute(::CurlEasy, ::String, Vector{String}; url::String, data_handler::Function, header_handler::Function)  → (CURLCode, Int64, String)

Execute a CurlEasy handle optionally passing in a requestBody (for POSTs), any HTTP request headers, a request URL, and handlers for response data and headers.

In its first form this method accepts the data_handler as the first argument allowing you to use curl_execute(curl) do data ... end to handle the data. In this case, response headers are ignored.

In its second form, both data and header handlers are passed in as keyword arguments. If not specified, then default handlers are set up that write to userdata[:databuffer] and userdata[:responseHeaders] respectively. You may explicitly set the handler to nothing to avoid handling data or headers. This can have a small improvement in memory utilization.

curl_execute(::CurlMulti) → CURLMcode

Executes all pending CurlEasy attached to the CurlMulti handle and returns a CURLMcode indicating success or failure.

In most cases, this function should return CURLM_OK even if there were failures in individual transfers. Each CurlEasy handle will have userdata[:http_status] set and userdata[:errormessage] will be set in case of an error.

This function will print errors or warnings to the Logger for unexpected states. File a bug if you see any of these.

Default header callback that puts the current header as a crlf terminate String onto a Channel passed in via curl_easy_setopt(CURLOPT_HEADERDATA).

This callback is called by curl when header data is available to be read and is set up in curl_setup_request

Run either curl_easy_perform or curl_multi_perform depending on the type of handle passed in.

curl_response_status(::CurlEasy) → Int64

Get the HTTP status code of the most recent response from the CurlEasy object.

Prepare a CurlEasy object for making a request.

• Adds the requestBody and a corresponding Content-Length
• Adds headers
• If data_channel or header_channel are set, then sets up a default WRITE/HEADER callback that writes to that Channel
• If url is set, sets the request URL

Setup the request object and response handlers in preparation to execute a request.

When using the CurlEasy interface, this method is called internally by curl_execute, however when using the CurlMulti interface, it is necessary to call this on every CurlEasy handle added to the CurlMulti handle.

This method allows you to set up your own response data and header handlers that receive streamed data. If you do not pass in a handler, default handlers will be set up that write binary data as bytes (Vector{UInt8}) to curl.userdata[:databuffer] and an array of String response headers (Vector{String}) to curl.userdata[:responseHeaders].

Arguments

curl::CurlEasy : The CurlEasy handle to operate on

requestBody::String : Any request body text that should be passed on to the server. Typically used for POST requests. Leave this as an empty String to skip. This is passed as-is to curl_setup_request.

headers::Vector{String} = String[] : Any request headers that should be passed on to the server as part of the request. Headers SHOULD be of the form key: value. Consult RFC 2616 section 4.2 for more details on HTTP request headers.

Keyword Arguments

data_handler::Union{Function, Nothing} = <default> : A function to handle any response Body data. This function should accept a single argument of type Vector{UInt8}. Its return value will be ignored. If not specified, a default handler will be used. Set this explicitly to nothing to disable handling of HTTP response body data.

data_handler::Union{Function, Nothing} = <default> : A function to handle any response Header data. This function should accept a single argument of type String. Its return value will be ignored. If not specified, a default handler will be used. Set this explicitly to nothing to disable handling of HTTP response header data.

url::AbstractString="" : The URL to use for this request. This permanently overrides the url passed in to the CurlEasy constructor. If not specified, then the previous value of the CurlEasy's url is reused.

Returns

The CurlEasy object.

curl_url_escape(::CurlEasy, ::String) → String
curl_url_escape(::String) → String

Use curl to do URL escaping

Default write callback that puts the data stream as a Vector{UInt8} onto a Channel passed in via curl_easy_setopt(CURLOPT_WRITEDATA).

This callback is called by curl when data is available to be read and is set up in curl_setup_request

Set the default user agent string to use for all requests. Set this to nothing to disable setting the user agent string.

Setup a Channel for the default response handlers to write to.

Cleanup everything created by the CurlEasy constructor. See the upstream docs for more details.

Clone a CurlEasy handle. See the upstream docs for more details.

URL escape a Julia string using a CurlEasy handle to make it safe for use as a URL. See the upstream docs

The return value is a Julia string with memory owned by Julia, so there's no risk of leaking memory.

Perform a CurlEasy transfer synchronously. See the upstream docs for more details.

Set options for the CurlEasy handle. See the upstream docs for all possible options, and links to documentation for each option.

Adds a CurlEasy handle to the CurlMulti pool. See the upstream docs

Cleanup everything created by the CurlMulti constructor. See the upstream docs for more details.

Perform all CurlEasy transfers attached to a CurlMulti handle asynchronously. See the upstream docs for more details.

Remove a CurlEasy handle from the CurlMulti pool. See the upstream docs. Pass in either the CurlEasy handle or its CurlHandle.uuid.