libcurls, which turned out to be the 25-nov-2006 change which treats HTTP
responses without Content-Length or chunked encoding as without bodies. We
now added the conditional that the above mentioned response is only without
body if the response is HTTP 1.1.
when CURLM_CALL_MULTI_PERFORM is returned from curl_multi_socket*/perform,
to make applications using only curl_multi_socket() to properly function
when adding easy handles "on the fly". Bug report and test app provided by
Michael Wallner.
since it then inits libgcrypt and libgcrypt is being evil and EXITS the
application if it fails to get a fine random seed. That's really not a nice
thing to do by a library.
been removed from a multi handle, and then fixed another flaw that prevented
curl_easy_duphandle() to work even after the first fix - the handle was
still marked as using the multi interface.
was 16385 bytes (16K+1) and it turned out we didn't properly always "suck
out" all data from libssh2. The effect being that libcurl would hang on the
socket waiting for data when libssh2 had in fact already read it all...
the CURLOPT_RESUME_FROM or CURLOPT_RANGE options and an existing connection
in the connection cache is closed to make room for the new one when you call
curl_easy_perform(). It would then wrongly free range-related data in the
connection close funtion.
identifying a double-free problem in the SSL-dealing layer, telling GnuTLS to
free NULL credentials on closedown after a failure and a bad #ifdef for NSS
when closing down SSL.
Curl_socket_ready(), Curl_poll() and Curl_select() when these are called
with a zero timeout or a timeout value indicating a blocking call should
be performed.
These unnecessary calls to gettimeofday() got introduced in 7.16.2 when
fixing 'timeout would restart when signal caught while awaiting socket
events' on 20 March 2007.
- Move some loop breaking logic from the while clause into the loop,
avoiding compiler warning 'assignment within conditional expression'
function that deprecates the curl_multi_socket() function. Using the new
function the application tell libcurl what action that was found in the
socket that it passes in. This gives a significant performance boost as it
allows libcurl to avoid a call to poll()/select() for every call to
curl_multi_socket*().
by letting configure check for setmode and ifdef on HAVE_SETMODE. NOTE: non-
configure platforms that havve setmode() needs their hard-coded config.h files
fixed. I fixed the src/config-win32.h.
or Curl_poll() with a non-zero timeout both functions would restart the
specified timeout. This could even lead to the extreme case that if a
signal arrived with a frecuency lower to the specified timeout neither
function would ever exit.
Added experimental symbol definition check CURL_ACKNOWLEDGE_EINTR in
Curl_select() and Curl_poll(). When compiled with CURL_ACKNOWLEDGE_EINTR
defined both functions will return as soon as a signal is caught. Use it
at your own risk, all calls to these functions in the library should be
revisited and checked before fully supporting this feature.
more frequently allowing same calling frecuency for the client progress
callback, while keeping the once a second frecuency for speed calculations
and internal display of the transfer progress.
makefiles that are included in the source release archives, generated from
the Makefile.vc6 files by the maketgz script. I also modified the root
Makefile to have a VC variable that defaults to vc6 but can be overridden to
allow it to be used for vc8 as well. Like this:
nmake VC=vc8 vc
server through a proxy and have the remote https server port set using the
CURLOPT_PORT option, protocol gets reset to http from https after the first
request.
User defined URL was modified internally by libcurl and subsequent reuse of
the easy handle may lead to connection using a different protocol (if not
originally http).
I found that libcurl hardcoded the protocol to "http" when it tries to
regenerate the URL if CURLOPT_PORT is set. I tried to fix the problem as
follows and it's working fine so far
fixing some bugs:
o Don't mix GET and POST requests in a pipeline
o Fix the order in which requests are dispatched from the pipeline
o Fixed several curl bugs with pipelining when the server is returning
chunked encoding:
* Added states to chunked parsing for final CRLF
* Rewind buffer after parsing chunk with data remaining
* Moved chunked header initializing to a spot just before receiving
headers
the multi interface and connection re-use that could make a
curl_multi_remove_handle() ruin a pointer in another handle.
The second problem was less of an actual problem but more of minor quirk:
the re-using of connections wasn't properly checking if the connection was
marked for closure.
to the debug callback.
- Shmulik Regev added CURLOPT_HTTP_CONTENT_DECODING and
CURLOPT_HTTP_TRANSFER_DECODING that if set to zero will disable libcurl's
internal decoding of content or transfer encoded content. This may be
preferable in cases where you use libcurl for proxy purposes or similar. The
command line tool got a --raw option to disable both at once.
and CURLOPT_CONNECTTIMEOUT_MS that, as their names should hint, do the
timeouts with millisecond resolution instead. The only restriction to that
is the alarm() (sometimes) used to abort name resolves as that uses full
seconds. I fixed the FTP response timeout part of the patch.
Internally we now count and keep the timeouts in milliseconds but it also
means we multiply set timeouts with 1000. The effect of this is that no
timeout can be set to more than 2^31 milliseconds (on 32 bit systems), which
equals 24.86 days. We probably couldn't before either since the code did
*1000 on the timeout values on several places already.
fail since they used "1 feb 2007"...
- Manfred Schwarb reported that socks5 support was broken and help us pinpoint
the problem. The code now tries harder to use httproxy and proxy where
apppropriate, as not all proxies are HTTP...
ordinary curl command line, and you will get a libcurl-using source code
written to the file that does the equivalent operation of what your command
line operation does!
doing an FTP transfer is removed from a multi handle before completion. The
fix also fixed the "alive counter" to be correct on "premature removal" for
all protocols.
non-ASCII platforms. It does add some complexity, most notably with more
#ifdefs, but I want to see this supported added and I can't see how we can
add it without the extra stuff added.
curl that uses the new CURLOPT_FTP_SSL_CCC option in libcurl. If enabled, it
will make libcurl shutdown SSL/TLS after the authentication is done on a
FTP-SSL operation.
downloaded data in two buffers, just to be able to deal with a special HTTP
pipelining case. That is now only activated for pipelined transfers. In
Matt's case, it showed as a considerable performance difference,
(http://curl.haxx.se/bug/view.cgi?id=1603712) (known bug #36) --limit-rate
(CURLOPT_MAX_SEND_SPEED_LARGE and CURLOPT_MAX_RECV_SPEED_LARGE) are broken
on Windows (since 7.16.0, but that's when they were introduced as previous
to that the limiting logic was made in the application only and not in the
library). It was actually also broken on select()-based systems (as apposed
to poll()) but we haven't had any such reports. We now use select(), Sleep()
or delay() properly to sleep a while without waiting for anything input or
output when the rate limiting is activated with the easy interface.
get confused and not acknowledge the 'no_proxy' variable properly once it
had used the proxy and you re-used the same easy handle. I made sure the
proxy name is properly stored in the connect struct rather than the
sessionhandle/easy struct.
(http://curl.haxx.se/bug/view.cgi?id=1618359) and subsequently provided a
patch for it: when downloading 2 zero byte files in a row, curl 7.16.0
enters an infinite loop, while curl 7.16.1-20061218 does one additional
unnecessary request.
Fix: During the "Major overhaul introducing http pipelining support and
shared connection cache within the multi handle." change, headerbytecount
was moved to live in the Curl_transfer_keeper structure. But that structure
is reset in the Transfer method, losing the information that we had about
the header size. This patch moves it back to the connectdata struct.
something went wrong like it got a bad response code back from the server,
libcurl would leak memory. Added test case 538 to verify the fix.
I also noted that the connection would get cached in that case, which
doesn't make sense since it cannot be re-use when the authentication has
failed. I fixed that issue too at the same time, and also that the path
would be "remembered" in vain for cases where the connection was about to
get closed.
(http://curl.haxx.se/bug/view.cgi?id=1603712) which is about connections
getting cut off prematurely when --limit-rate is used. While I found no such
problems in my tests nor in my reading of the code, I found that the
--limit-rate code was severly flawed (since it was moved into the lib, since
7.15.5) when used with the easy interface and it didn't work as documented so
I reworked it somewhat and now it works for my tests.
passing a curl_off_t argument to the Curl_read_rewind() function which takes
an size_t argument. Curl_read_rewind() also had debug code left in it and it
was put in a different source file with no good reason when only used from
one single spot.
no code present in the library that receives the option. Since it was not
possible to use, we know that no current users exist and thus we simply
removed it from the docs and made the code always use the default path of
the code.
(http://curl.haxx.se/bug/view.cgi?id=1604956) which identified setting
CURLOPT_MAXCONNECTS to zero caused libcurl to SIGSEGV. Starting now, libcurl
will always internally use no less than 1 entry in the connection cache.
(http://curl.haxx.se/bug/view.cgi?id=1230118) curl_getdate() did not work
properly for all input dates on Windows. It was mostly seen on some TZ time
zones using DST. Luckily, Martin also provided a fix.
(http://curl.haxx.se/bug/view.cgi?id=1600447) in which he noted that active
FTP connections don't work with the multi interface. The problem is here that
the multi interface state machine has a state during which it can wait for the
data connection to connect, but the active connection is not done in the same
step in the sequence as the passive one is so it doesn't quite work for
active. The active FTP code still use a blocking function to allow the remote
server to connect.
The fix (work-around is a better word) for this problem is to set the
boolean prematurely that the data connection is completed, so that the "wait
for connect" phase ends at once.
HTTP upload was disconnected:
"What appears to be happening is that my system (Linux 2.6.17 and 2.6.13) is
setting *only* POLLHUP on poll() when the conditions in my previous mail
occur. As you can see, select.c:Curl_select() does not check for POLLHUP. So
basically what was happening, is poll() was returning immediately (with
POLLHUP set), but when Curl_select() looked at the bits, neither POLLERR or
POLLOUT was set. This still caused Curl_readwrite() to be called, which
quickly returned. Then the transfer() loop kept continuing at full speed
forever."
header in a third, not suppported by libcurl, format and we agreed that we
could make the parser more forgiving to accept all the three found
variations.
responded with a single status line and no headers nor body. Starting now, a
HTTP response on a persistent connection (i.e not set to be closed after the
response has been taken care of) must have Content-Length or chunked
encoding set, or libcurl will simply assume that there is no body.
To my horror I learned that we had no less than 57(!) test cases that did bad
HTTP responses like this, and even the test http server (sws) responded badly
when queried by the test system if it is the test system. So although the
actual fix for the problem was tiny, going through all the newly failing test
cases got really painful and boring.
KNOWN_BUGS #25, which happens when a proxy closes the connection when
libcurl has sent CONNECT, as part of an authentication negotiation. Starting
now, libcurl will re-connect accordingly and continue the authentication as
it should.
case when 401 or 407 are returned, *IF* no auth credentials have been given.
The CURLOPT_FAILONERROR option is not possible to make fool-proof for 401
and 407 cases when auth credentials is given, but we've now covered this
somewhat more.
You might get some amounts of headers transferred before this situation is
detected, like for when a "100-continue" is received as a response to a
POST/PUT and a 401 or 407 is received immediately afterwards.
Added test 281 to verify this change.
to the CURLOPT_DEBUGFUNCTION callback has been fixed (it was erroneously
included as part of the header). A message was also added to the
command line tool to show when data is being sent, enabled when
--verbose is used.
and while doing so it became apparent that the current timeout system for
the socket API really was a bit awkward since it become quite some work to
be sure we have the correct timeout set.
Jeff then provided the new CURLMOPT_TIMERFUNCTION that is yet another
callback the app can set to get to know when the general timeout time
changes and thus for an application like hiperfifo.c it makes everything a
lot easier and nicer. There's a CURLMOPT_TIMERDATA option too of course in
good old libcurl tradition.
would crash if a bad function sequence was used when shutting down after
using the multi interface (i.e using easy_cleanup after multi_cleanup) so
precautions have been added to make sure it doesn't any more - test case 529
was added to verify.
handle that is part of a multi handle first removes the handle from the
stack.
- Added CURLOPT_SSL_SESSIONID_CACHE and --no-sessionid to disable SSL
session-ID re-use on demand since there obviously are broken servers out
there that misbehave with session-IDs used.
while not fixing things very nicely, it does make the SOCKS5 proxy
connection slightly better as it now acknowledges the timeout for connection
and it no longer segfaults in the case when SOCKS requires authentication
and you did not specify username:password.
send the whole request at once, even though the Expect: header was disabled
by the application. An effect of this change is also that small (< 1024
bytes) POSTs are now always sent without Expect: header since we deem it
more costly to bother about that than the risk that we send the data in
vain.
CURLOPT_NOBODY is set true. PREQUOTE is then run roughly at the same place
in the command sequence as it would have run if there would've been a
transfer.
we did wrong and patched it: When nodes were removed from the splay tree,
and we didn't properly remove it from the splay tree when an easy handle was
removed from a multi stack and thus we could wrongly leave a node in the
splay tree pointing to (bad) memory.
the crash was that libcurl internally was a bit confused about who owned the
DNS cache at all times so if you created an easy handle that uses a shared
DNS cache and added that to a multi handle it would crash. Now we keep more
careful internal track of exactly what kind of DNS cache each easy handle
uses: None, Private (allocated for and used only by this single handle),
Shared (points to a cache held by a shared object), Global (points to the
global cache) or Multi (points to the cache within the multi handle that is
automatically shared between all easy handles that are added with private
caches).
CURLOPT_MAX_RECV_SPEED_LARGE that limit tha maximum rate libcurl is allowed
to send or receive data. This kind of adds the the command line tool's
option --limit-rate to the library.
The rate limiting logic in the curl app is now removed and is instead
provided by libcurl itself. Transfer rate limiting will now also work for -d
and -F, which it didn't before.
autotools project, which optionally (default=yes) uses libcurl on a system
without a (usable) libcurl installation, but not specifying
`--without-libcurl', configure determines correctly that no libcurl is
available, however, the LIBCURL variable gets expanded to `LIBCURL = -lcurl'
in the resulting Makefiles.
David Shaw fixed the flaw.
multi stack and that easy handle had already been used to do one or more
easy interface transfers, as then the code threw away the previously used
DNS cache without properly freeing it.
(http://curl.haxx.se/bug/view.cgi?id=1481217), with follow-ups by Michele Bini
and David Byron. libcurl previously wrongly used GetLastError() on windows to
get error details after socket-related function calls, when it really should
use WSAGetLastError() instead.
When changing to this, the former function Curl_ourerrno() is now instead
called Curl_sockerrno() as it is necessary to only use it to get errno from
socket-related functions as otherwise it won't work as intended on Windows.
(http://curl.haxx.se/bug/view.cgi?id=1480821) He found and identified a
problem with how libcurl dealt with GnuTLS and a case where gnutls returned
GNUTLS_E_AGAIN indicating it would block. It would then return an unexpected
return code, making Curl_ssl_send() confuse the upper layer - causing random
28 bytes trash data to get inserted in the transfered stream.
The proper fix was to make the Curl_gtls_send() function return the proper
return codes that the callers would expect. The Curl_ossl_send() function
already did this.
(http://curl.haxx.se/mail/lib-2006-02/0154.html) by adding the NTLM hash
function in addition to the LM one and making some other adjustments in the
order the different parts of the data block are sent in the Type-2 reply.
Inspiration for this work was taken from the Firefox NTLM implementation.
I edited the existing 21(!) NTLM test cases to run fine with these news. Due
to the fact that we now properly include the host name in the Type-2 message
the test cases now only compare parts of that chunk.
--enable-debug, as then curl used free() on memory allocated both with
normal malloc() and with libcurl-provided functions, when the latter MUST be
freed with curl_free() in debug builds.
with the multi interface and multi-part formposts. The fix from February
22nd could make the Curl_done() function get called twice on the same
connection and it was not designed for that and thus tried to call free() on
an already freed memory area!
callback" error message so I've now made the setting of that callback not be
as critical as before. The function is only used for additional loggging/
trace anyway so a failure just means slightly less data. It should still be
able to proceed and connect fine to the server.
(http://curl.haxx.se/bug/view.cgi?id=1431750) helped me identify and fix two
different but related bugs:
1) Removing an easy handle from a multi handle before the transfer is done
could leave a connection in the connection cache for that handle that is
in a state that isn't suitable for re-use. A subsequent re-use could then
read from a NULL pointer and segfault.
2) When an easy handle was removed from the multi handle, there could be an
outstanding c-ares DNS name resolve request. When the response arrived,
it caused havoc since the connection struct it "belonged" to could've
been freed already.
Now Curl_done() is called when an easy handle is removed from a multi handle
pre-maturely (that is, before the transfer was complteted). Curl_done() also
makes sure to cancel all (if any) outstanding c-ares requests.
type to the already provided type CURLPROXY_SOCKS4.
I added a --socks4 option that works like the current --socks5 option but
instead use the socks4 protocol.
an app can use to let libcurl only connect to a remote host and then extract
the socket from libcurl. libcurl will then not attempt to do any transfer at
all after the connect is done.
with re-used FTP connections. If the second request on the same connection was
set not to fetch a "body", libcurl could get confused and consider it an
attempt to use a dead connection and would go acting mighty strange.
curl tool with --local-port. Plain and simply set the range of ports to bind
the local end of connections to. Implemented on to popular demand.
Not extensively tested. Please let me know how it works.
connection setup as a follow-redirect. It turns out 1) this fails when a FTP
connection is re-setup and 2) it does make the max-redirs counter behave
wrong. This fix was not verified since the reporter vanished, but I believe
this is the right fix nonetheless.
even after EPSV returned a positive response code, if libcurl failed to
connect to the port number the EPSV response said. Obviously some people are
going through protocol-sensitive firewalls (or similar) that don't understand
EPSV and then they don't allow the second connection unless PASV was
used. This also called for a minor fix of test case 238.
into a counter, and thus you can now do multiple curl_global_init() and you
are then supposed to do the same amount of calls to curl_global_cleanup().
Bryan also updated the docs accordingly.
given subdirs, libcurl would still "remember" the full path as if it is the
current directory libcurl is in so that the next curl_easy_perform() would
get really confused if it tried the same path again - as it would not issue
any CWD commands at all, assuming it is already in the "proper" dir.
Starting now, a failed CWD command sets a flag that prevents the path to be
"remembered" after returning.
http://www.greenend.org.uk/rjk/2001/06/poll.html and further tests by Eugene
Kotlyarov, we now know that cygwin's poll returns only POLLHUP on remote
connection closure so we check for that case (too) and re-enable poll for
cygwin builds.
The LDAP code in libcurl can't handle LDAP servers of LDAPv3 nor binary
attributes in LDAP objects. So, I made a quick patch to address these
problems.
The solution is simple: if we connect to an LDAP server, first try LDAPv3
(which is the preferred protocol as of now) and then fall back to LDAPv2.
In case of binary attributes, we first convert them to base64, just like the
openldap client does. It uses ldap_get_values_len() instead of
ldap_get_values() to be able to retrieve binary attributes correctly. I
defined the necessary LDAP macros in lib/ldap.c to be able to compile
libcurl without the presence of libldap
(http://curl.haxx.se/bug/view.cgi?id=1337723) that curl could not upload
binary data from stdin on Windows if the data contained control-Z (hex 1a)
since that is treated as end-of-file when read in text mode. Gisle Vanem
pointed out the fix, and I made both -T and --data-binary take advantage of
it.
in the man page, curl would send an invalid HTTP Range: header. The correct
way would be to use "-r [number]-" or even "-r -[number]". Starting now,
curl will warn if this is discovered, and automatically append a dash to the
range before passing it to libcurl.
(wrongly) sends *two* WWW-Authenticate headers for Digest. While this should
never happen in a sane world, libcurl previously got into an infinite loop
when this occurred. Dave added test 273 to verify this.
reported, the define is used by the configure script and is assumed to use
the 0xYYXXZZ format. This made "curl-config --vernum" fail in the 7.15.0
release version.
(http://curl.haxx.se/bug/view.cgi?id=1299181) that identified a silly problem
with Content-Range: headers with the 'bytes' keyword written in a different
case than all lowercase! It would cause a segfault!
for Windows, that could lead to an Access Violation when the multi interface
was used due to an issue with how the resolver thread was and was not
terminated.
from the command line tool with --ignore-content-length. This will make it
easier to download files from Apache 1.x (and similar) servers that are
still having problems serving files larger than 2 or 4 GB. When this option
is enabled, curl will simply have to wait for the server to close the
connection to signal end of transfer. I wrote test case 269 that runs a
simple test that this works.
that made curl run fine in his end. The key was to make sure we do the
SSL/TLS negotiation immediately after the TCP connect is done and not after
a few other commands have been sent like we did previously. I don't consider
this change necessary to obey the standards, I think this server is pickier
than what the specs allow it to be, but I can't see how this modified
libcurl code can add any problems to those who are interpreting the
standards more liberally.
CURLOPT_COOKIEFILE), add a cookie (with CURLOPT_COOKIELIST), tell it to
write the result to a given cookie jar and then never actually call
curl_easy_perform() - the given file(s) to read was never read but the
output file was written and thus it caused a "funny" result.
- While doing some tests for the bug above, I noticed that Firefox generates
large numbers (for the expire time) in the cookies.txt file and libcurl
didn't treat them properly. Now it does.
zone name of a daylight savings time was used. For example, PDT vs PDS. This
flaw was introduced with the new date parser (11 sep 2004 - 7.12.2).
Fortunately, no web server or cookie string etc should be using such time
zone names thus limiting the effect of this bug.
HTTP proxy if an FTP URL was given. libcurl now properly switches to pure HTTP
internally when an HTTP proxy is used, even for FTP URLs. The problem would
also occur with other multi-pass auth methods.
seems the Windows (MSVC) libc time functions may return data one hour off if
TZ is not set and automatic DST adjustment is enabled. This made
curl_getdate() return wrong value, and it also concerned internal cookie
expirations etc.
fix the CONNECT authentication code with multi-pass auth methods (such as
NTLM) as it didn't previously properly ignore response-bodies - in fact it
stopped reading after all response headers had been received. This could
lead to libcurl sending the next request and reading the body from the first
request as response to the second request. (I also renamed the function,
which wasn't strictly necessary but...)
The best fix would to once and for all make the CONNECT code use the
ordinary request sending/receiving code, treating it as any ordinary request
instead of the special-purpose function we have now. It should make it
better for multi-interface too. And possibly lead to less code...
Added test case 265 for this. It doesn't work as a _really_ good test case
since the test proxy is too stupid, but the test case helps when running the
debugger to verify.
address was not possible to use. It is now, but requires it written
RFC2732-style, within brackets - which incidently is how you enter numerical
IPv6 addresses in URLs. Test case 263 added to verify.
--trace, --trace-ascii and --verbose output. I also made the '>' display
separate each line on the linefeed so that HTTP requests etc look nicer in the
-v output.
file got a Last-Modified: header written to the data stream, corrupting the
actual data. This was because some conditions from the previous FTP code was
not properly brought into the new FTP code. I fixed and I added test case 520
to verify. (This bug was introduced in 7.13.1)
on the remote side. This then converts the operation to an ordinary STOR
upload. This was requested/pointed out by Ignacio Vazquez-Abrams.
It also proved (and I fixed) a bug in the newly rewritten ftp code (and
present in the 7.13.1 release) when trying to resume an upload and the servers
returns an error to the SIZE command. libcurl then loops and sends SIZE
commands infinitely.
requested data from a host and then followed a redirect to another
host. libcurl then didn't use the proxy-auth properly in the second request,
due to the host-only check for original host name wrongly being extended to
the proxy auth as well. Added test case 233 to verify the flaw and that the
fix removed the problem.
that picks NTLM. Thanks to David Byron letting me test NTLM against his
servers, I could quickly repeat and fix the problem. It turned out to be:
When libcurl POSTs without knowing/using an authentication and it gets back a
list of types from which it picks NTLM, it needs to either continue sending
its data if it keeps the connection alive, or not send the data but close the
connection. Then do the first step in the NTLM auth. libcurl didn't send the
data nor close the connection but simply read the response-body and then sent
the first negotiation step. Which then failed miserably of course. The fixed
version forces a connection if there is more than 2000 bytes left to send.
Daniel Stenberg
Yang Tse
Dan Fandrich
Linus Nielsen Feltzing