WEBrick::HTTPProxyServer implementes HTTP proxy using
WEBrick and Net::HTTP.
WEBrick accepts HTTP/1.0 clients and
Net::HTTP uses always HTTP/1.1.
However HTTP/1.1 supports chunked transfer coding HTTP/1.0 doesn't.
Chunked transfer coding doesn't require that
content-length before the content is sent.
But non-chunked transfer coding require content-length before
the content is sent.
So, when HTTP/1.0 clients connects WEBrick::HTTPProxyServer and
origin server returns chunked response,
WEBrick::HTTPProxyServer needs to store whole content to
know the length of it.
This patch do it using tempfile.
Remove extraneous spaces after the status code that is
non-compliant with RFC, i.e `HTTP 200 OK `, to unnecessary
confusion for WEBrick users, by a risk that WEBrick instances in
the wild will have server responses flagged as suspicious or
malicious due to a similar bug in [Cobalt Strike
misconfiguration].
Reported by Matt Tennis <mtennis@paloaltonetworks.com>
[Cobalt Strike misconfiguration]: https://blog.fox-it.com/2019/02/26/identifying-cobalt-strike-team-servers-in-the-wild/
So that a customized HTTPServer subclass can use it's own
Request/Response classes.
To apply the override, make a subclass of WEBrick::HTTPServer
and override the
`create_request_and_response(with_webrick_config)` method. The
method should return an Array of [request, response].
To check whether the Server supports this method (i.e. when
using older versions of WEBrick when needing this
functionality), you can ask the server if it responds to the
method
server.respond_to?(:create_request_and_response)
This is backportable.
[ruby-core:69604] [Feature #11266]
From: Julik Tarkhanov <me@julik.nl>
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66452 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
OpenSSL complains abour our keys being small and weak :<
Make them big and strong with 2048-bit RSA keys and SHA256 digests
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66152 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
The values of @header are expected to be all strings;
WEBrick::HTTPResponse::[]=(key, val) explicitly converts the second
argument to a string and assigns it to @header hash.
However, there were some points in WEBrick internal code that assigns
non-String to @header. This change fixes the issues.
The values are checked by `header_value =~ /\r\n/` in check_header.
The type confusion caused conflict with removal of `Object#=~`
[Feature #15231].
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65984 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Based on patch by akr [ruby-core:88477], use Tempfile.create
to avoid unnecessary unlink call. Unlike akr's original patch,
this does not change the return value of flush.
Thanks-to: Tanaka Akira <akr@fsij.org>
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64363 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
They are assigned automatically when pushing gem file to rubygems.org.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64213 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
This adds a password_hash keyword argument to
WEBrick::HTTPAuth::Htpasswd#initialize. If set to :bcrypt, it
will create bcrypt hashes instead of crypt hashes, and will
raise an exception if the .htpasswd file uses crypt hashes.
If :bcrypt is used, then instead of calling
BasicAuth.make_passwd (which uses crypt),
WEBrick::HTTPAuth::Htpasswd#set_passwd will set the bcrypt
password directly. It isn't possible to change the
make_passwd API to accept the password hash format, as that
would break configurations who use Htpasswd#auth_type= to set
a custom auth_type.
This modifies WEBrick::HTTPAuth::BasicAuth to handle checking
both crypt and bcrypt hashes.
There are commented out requires for 'string/crypt', to handle
when String#crypt is deprecated and the undeprecated version is
moved to a gem.
There is also a commented out warning for the case when
the password_hash keyword is not specified and 'string/crypt'
cannot be required. I think the warning makes sense to nudge
users to using bcrypt.
I've updated the tests to test nil, :crypt, and :bcrypt values
for the password_hash keyword, skipping the bcrypt tests if the
bcrypt library cannot be required.
[ruby-core:88111] [Feature #14940]
From: Jeremy Evans <code@jeremyevans.net>
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64060 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Prevents response splitting and HTML injection attacks in
poorly-written applications which blindly pass along user input
in redirects.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63964 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* lib/webrick/httpresponse.rb (send_body_io): use seek if NotImplementedError
is raised in IO.copy_stream with offset.
* lib/webrick/httpservlet/filehandler.rb (multipart_body): ditto.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63008 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Original patch by tenderlove (with minor style adjustments).
* lib/webrick/httpresponse.rb (send_header): call check_header
(check_header): raise on embedded CRLF in header value
* test/webrick/test_httpresponse.rb
(test_prevent_response_splitting_headers): new test
* (test_prevent_response_splitting_cookie_headers): ditto
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62968 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Reading entire request or response bodies into memory can lead
to trivial denial-of-service attacks. Introduce Fibers in both
cases to allow streaming.
WEBrick::HTTPRequest gains a new body_reader method to prepare
itself as a source for IO.copy_stream. This allows the
WEBrick::HTTPRequest object to be used as the
Net::HTTPGenericRequest#body_stream= arg for Net::HTTP.
For HTTP proxy response bodies, we also use a Fiber to
to make the HTTP request and read the response body.
* lib/webrick/httprequest.rb (body_reader): new method
(readpartial): ditto
* lib/webrick/httpproxy.rb (perform_proxy_request): use Fiber
to stream response body
(do_GET, do_HEAD): adjust call
(do_POST): adjust call and supply body_reader
* test/webrick/test_httprequest.rb (test_chunked): test
for IO.copy_stream compatibility
* test/webrick/test_httpproxy.rb (test_big_bodies): new test
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62966 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
WARNING! WARNING! WARNING! LIKELY BROKEN CHANGE
Pass a proc to WEBrick::HTTPRequest#body to avoid reading a
potentially large request body into memory during
authentication.
WARNING! this will break apps completely which want to do
something with the body besides calculating the MD5 digest
of it.
Also, keep in mind that probably nobody uses "auth-int".
Servers such as Apache, lighttpd, nginx don't seem to
support it; nor does curl when using POST/PUT bodies;
and we didn't have tests for it until now...
* lib/webrick/httpauth/digestauth.rb (_authenticate): stream req.body
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62965 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
While WEBrick::HTTPRequest#body provides a Proc interface
for streaming large request bodies, clients must not force
the server to use an excessively large chunk size.
* lib/webrick/httprequest.rb (read_chunk_size): limit each
read and block.call to :InputBufferSize in config.
* test/webrick/test_httpserver.rb (test_big_chunks): new test
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62963 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
"BadRequest" alone does not resolve correctly, it is in the
HTTPStatus namespace.
* lib/webrick/httprequest.rb (read_chunked): use correct exception
* test/webrick/test_httpserver.rb (test_eof_in_chunk): new test
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62962 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
WEBrick::HTTPRequest#body can be passed a block to process the
body in chunks. Use this feature to avoid building a giant
string in memory.
* lib/webrick/httpservlet/cgihandler.rb (do_GET):
avoid reading entire request body into memory
(do_POST is aliased to do_GET, so it handles bodies)
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62961 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
We use the same 112 KB limit started (AFAIK) by Mongrel, Thin,
and Puma to prevent malicious users from using up all the memory
with a single request. This also limits the damage done by
excessive ranges in multipart Range: requests.
Due to the way we rely on IO#gets and the desire to keep
the code simple, the actual maximum header may be 4093 bytes
larger than 112 KB, but we're splitting hairs at that point.
* lib/webrick/httprequest.rb: define MAX_HEADER_LENGTH
(read_header): raise when headers exceed max length
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62960 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Use the new Proc response body feature to generate a multipart
range response dynamically. We use a flat array to minimize
object overhead as much as possible; as many ranges may fit
into an HTTP request header.
* lib/webrick/httpservlet/filehandler.rb (multipart_body): new method
(make_partial_content): use multipart_body
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62959 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
The .write method needs to return the number of bytes written
to avoid confusing IO.copy_stream.
* lib/webrick/httpresponse.rb (ChunkedWrapper#write): return bytes written
(ChunkedWrapper#<<): return self
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62958 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Content-Length is ignored by WEBrick::HTTPResponse even if we
calculate it, so instead we chunk responses to HTTP/1.1 clients
and terminate HTTP/1.0 connections.
* lib/webrick/httpservlet/filehandler.rb (make_partial_content):
quiet warning
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62957 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
This is also compatible with range responses generated
by Rack::File (tested with rack 2.0.3).
* lib/webrick/httpresponse.rb (send_body_io): use Content-Range
* lib/webrick/httpservlet/filehandler.rb (make_partial_content):
use File object for the single range case
* test/webrick/test_filehandler.rb (get_res_body): use send_body
to test result
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62955 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Remove the redundant _send_file method since its functionality
is unnecessary with IO.copy_stream. IO.copy_stream also allows
the use of sendfile under some OSes to speed up copies to
non-TLS sockets.
Testing with "curl >/dev/null" and "ruby -run -e httpd" to
read a 1G file over Linux loopback reveals a reduction from
around ~0.770 to ~0.490 seconds on the client side.
* lib/webrick/httpresponse.rb (send_body_io): use IO.copy_stream
(_send_file): remove
[Feature #14237]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62954 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
This will make the next change to use IO.copy_stream
easier-to-read. When we can drop Ruby 2.4 support in a few
years, this will allow us to use writev(2) with multiple
arguments for headers and chunked responses.
* lib/webrick/cgi.rb (write): new wrapper method
lib/webrick/httpresponse.rb: (send_header): use socket.write
(send_body_io): ditto
(send_body_string): ditto
(send_body_proc): ditto
(_write_data): ditto
(ChunkedWrapper#write): ditto
(_send_file): ditto
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62953 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
This release removes uses of Kernel#open to avoid unintended
behaviors and make future auditing easier. [Misc #14216]
6 changes since 1.4.1:
webrick: httpauth requires regular files
webrick/httpservlet/cgi_runner.rb: remove unnecessary open
webrick: WEBrick::Log requires path arg when given string
webrick/httpservlet/*handler: use File.open
webrick/httputils: note Kernel#open behavior
webrick/httpservelet/cgi_runner: avoid IO#reopen on pathname
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61443 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
IO#reopen seems to behave strangely on Win32 based on the logs
posted by Greg for [Bug #14420]. Lets try to fix the issue by
using File.open explicitly (but not Kernel#open).
* lib/webrick/httpservelet/cgi_runner: use File.open explicitly
[Bug #14220]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61409 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
I don't know who uses the load_mime_types method; but it is
conceivable that a user would want to read the results of a
command instead of reading a regular file to load MIME types.
None of the WEBrick-related code in Ruby or default/bundled gems
seems to rely on this method; but it is likely 3rd-party code does.
* lib/webrick/httputils.rb (load_mime_types): note Kernel#open behavior
[Misc #14216]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61402 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
This makes future code audits easier. None of these changes
fix realistic remote code execution vulnerabilities because
we stat(2) before attempting Kernel#open.
* lib/webrick/httpservlet/erbhandler.rb (do_GET): use File.open
* lib/webrick/httpservlet/filehandler.rb (do_GET): use File.open
(make_partial_content): ditto
[Misc #14216]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61401 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Allowing a user to specify "| command" via Kernel#open is
nonsensical since we never read from the resultant IO.
* lib/webrick/log.rb (initialize): replace Kernel#open with File.open
[Misc #14216]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61400 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
IO#reopen already takes string path names as well as IO objects
(but not "| command" strings)
This makes further auditing for inadvertant code execution
easier. There's no actual bugfix or behavior change here,
as no external data is passed to cgi_runner.rb.
* lib/webrick/httpservlet/cgi_runner.rb: remove Kernel#open call
[Misc #14216]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61398 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Be sure we do not try to open a pipe to read from, since we care
about mtime in all cases.
* lib/webrick/httpauth/htdigest.rb: use File.open
* lib/webrick/httpauth/htgroup.rb: ditto
* lib/webrick/httpauth/htpasswd.rb: ditto
[Misc #14216]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61397 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* lib/webrick/version.rb: change VERSION to 1.4.0
* lib/webrick/webrick.gemspec: require_relative for version,
update date and contact info
[Feature #13173]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61266 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Using ^ and $ in regexps means we can accidentally get fooled
by "%0a" in HTTP request paths being decoded to newline
characters. Use \A and \z to match beginning and end-of-string
respectively, instead.
Thanks to mame and hsbt for reporting.
* lib/webrick/httpserver.rb (MountTable#compile):
use \A and \z instead of ^ and $
* lib/webrick/httpserver.rb (MountTable#normalize): use \z instead of $
* test/webrick/test_httpserver.rb (test_cntrl_in_path): new test
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61197 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
If uplevel keyword is given, the warning message is prepended
with caller file and line information and the string "warning: ".
The use of the uplevel keyword makes Kernel#warn format output
similar to how rb_warn formats output.
This patch modifies net/ftp and net/imap to use Kernel#warn
instead of $stderr.puts or $stderr.printf, since they are used
for printing warnings.
This makes lib/cgi/core and tempfile use $stderr.puts instead of
warn for debug logging, since they are used for debug printing
and not for warning.
This does not modify bundler, rubygems, or rdoc, as those are
maintained outside of ruby and probably wish to remain backwards
compatible with older ruby versions.
rb_warn_m code is originally from nobu, but I've changed it
so that it only includes the path and lineno from uplevel
(not the method), and also prepends the string "warning: ",
to make it more similar to rb_warn.
From: Jeremy Evans code@jeremyevans.net
Signed-off-by: Urabe Shyouhei shyouhei@ruby-lang.org
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61155 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
We must to ensure the @status ivar is set to :Running before
running StartCallback, otherwise Webrick::Server#stop will not
change the @status to :Shutdown properly.
Note: I have not been able to reproduce the original issue but
understood at least part of the problem and fixed it with this
commit. However, the original reporter (Peak Xu) was still able
to reproduce the problem on 1.9.2 p180 on Windows, so I'm not
sure what else might be going on. Ruby threading and
synchronization primitives have changed a lot since 1.9.2, so
maybe that was fixed elsewhere.
* lib/webrick/server.rb: call StartCallback sooner [Bug #4841]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61083 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
I never knew "format" was a global method alias for "sprintf";
so it was confusing to me. Normally, one would use "sprintf"
since it's also available in many other languages, but
Integer#to_s avoids parsing a format string so it's less
bug-prone.
Furthermore, favor string interpolation over String#<< since it
is easier for the VM to optimize memory allocation (as in r60320).
Interpolation also reduces method calls and memory overhead
for inline method cache.
Finally, ensure we clear all short-lived buffers for body
responses. A similar change was made and measured for Net::*
in r58840 showing a large memory reduction on some workloads.
* webrick/httpresponse.rb (send_body_io): favor String#to_s,
reduce method calls for String#<<,
clear `buf' when done,
avoid extra String#bytesize calls
* (send_body_string): ditto
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@60586 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* lib/webrick/httpresponse.rb (send_body): call send_body_proc
(send_body_proc): new method
(class ChunkedWrapper): new class
* test/webrick/test_httpresponse.rb (test_send_body_proc): new test
(test_send_body_proc_chunked): ditto
[Feature #855]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@60584 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Thanks to MSP-Greg (Greg L) for helping with this.
* lib/webrick/server.rb (start_thread): properly fix non-local return
introduced in r60208 and r60210
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@60211 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Thanks to MSP-Greg (Greg L) for helping with this.
* lib/webrick/server.rb (start_thread): ignore ECONNRESET, ECONNABORTED,
EPROTO, and EINVAL on TLS negotiation errors the same way they
were ignored before r60172 in the accept_client method of the
main acceptor thread.
[Bug #14013] [Bug #14005]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@60208 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Tanaka Akira
Rob
Nobuyoshi Nakada
Hiroshi SHIBATA
normal
mame
marcandre
stomar
hsbt
ko1
naruse
usa
shyouhei