CURLE_OUT_OF_MEMORY when adding large header after update to 7.71.0 #6681
Description
I did this
An application I'm working on uses libcurl to make HTTP requests to a server and uses the following code pattern to make this request:
// setup
curl_easy_reset(m_curl_handle);
curl_easy_setopt(m_curl_handle, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_1_1);
curl_easy_setopt(m_curl_handle, CURLOPT_TIMEOUT, 0L);
curl_easy_setopt(m_curl_handle, CURLOPT_CONNECTTIMEOUT, 1L);
curl_easy_setopt(m_curl_handle, CURLOPT_NOSIGNAL, 1L);
curl_easy_setopt(m_curl_handle, CURLOPT_TCP_NODELAY, 1L);
curl_easy_setopt(m_curl_handle, CURLOPT_POST, 1L);
curl_easy_setopt(m_curl_handle, CURLOPT_READFUNCTION, read_data);
curl_easy_setopt(m_curl_handle, CURLOPT_WRITEFUNCTION, write_data);
curl_easy_setopt(m_curl_handle, CURLOPT_HEADERFUNCTION, write_header);
curl_easy_setopt(m_curl_handle, CURLOPT_PROXY, "");
curl_easy_setopt(m_curl_handle, CURLOPT_URL, url.c_str());
curl_slist* headers = nullptr;
headers = curl_slist_append(headers, "content-type: text/html");
headers = curl_slist_append(headers, ("large-header: " + up_to_1mb_string).c_str());
std::pair<std::string const&, size_t> ctx{payload, 0};
aws::http::response resp;
curl_easy_setopt(m_curl_handle, CURLOPT_WRITEDATA, &resp);
curl_easy_setopt(m_curl_handle, CURLOPT_HEADERDATA, &resp);
curl_easy_setopt(m_curl_handle, CURLOPT_READDATA, &ctx);
curl_easy_setopt(m_curl_handle, CURLOPT_HTTPHEADER, headers);
CURLcode curl_code = curl_easy_perform(m_curl_handle);
curl_slist_free_all(headers);
if (curl_code != CURLE_OK) {
// log error
}
We compile curl from source with:
./configure \
--prefix $INSTALL_PREFIX \
--disable-shared \
--enable-debug \ # added to debug this issue (logs below)
--without-ssl \
--with-pic \
--without-zlib && \
make && \
make install
Execution logs:
[DEBUG] [1614694131840] CURL DBG: STATE: INIT => CONNECT handle 0x14e9728; line 1771 (connection #-5000)
[DEBUG] [1614694131840] CURL DBG: Found bundle for host 127.0.0.1: 0x14f82b8 [serially]
[DEBUG] [1614694131840] CURL DBG: Re-using existing connection! (#0) with host 127.0.0.1
[DEBUG] [1614694131840] CURL DBG: Connected to 127.0.0.1 (127.0.0.1) port 9001 (#0)
[DEBUG] [1614694131840] CURL DBG: STATE: CONNECT => DO handle 0x14e9728; line 1825 (connection #0)
[DEBUG] [1614694131846] CURL DBG: multi_done
[DEBUG] [1614694131846] CURL DBG: Connection #0 to host 127.0.0.1 left intact
[DEBUG] [1614694131846] CURL DBG: Expire cleared (transfer 0x14e9728)
[DEBUG] [1614694131846] CURL returned error code 27 - Out of memory
I expected the following
Before version 7.71.0 (tested up to 7.75.0), this worked fine for headers with size up to 1mb but after the update I've started seeing CURLE_OUT_OF_MEMORY.
I'm not entirely sure what's changed in this version to blow up with OOM exceptions on large headers, but I suspect maybe aee277b as it's the closest change to the issue that I can see. Have not confirmed this however.
curl/libcurl version
7.71.0+
operating system
Amazon Linux 2
$ uname -a
Linux 331a706e46ef 4.19.121-linuxkit #1 SMP Tue Dec 1 17:50:32 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
I'd really appreciate guidance on this, thanks!