[Bug #14600]
* lib/rexml/parsers/xpathparser.rb: Fix a bug that "N-M" requires a space
before "-".
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63128 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Gem::TestCase normalizes each path in $LOAD_PATH, which deleted the flag
of @gem_prelude_index.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63100 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
[Bug #14600]
* lib/rexml/xpath_parser.rb: Fix a bug that "/child::node()" returns
XML declaration and text nodes out of root element.
* test/rexml/test_jaxen.rb: Enable more tests.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63088 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
[Bug #14600]
* lib/rexml/functions.rb: Fix a bug that "@attribute/parent" doesn't
return element of its attribute.
* test/rexml/test_jaxen.rb: Enable more tests.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63083 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
[Bug #14600]
* lib/rexml/functions.rb: Fix a bug that "name(node-set)" returns
element instead of element name.
* test/rexml/test_jaxen.rb: Enable more tests.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63079 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* lib/mkmf.rb: Improve the error message when ruby.h is missing,
to suggest installing separate packages. [Feature #14656]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63061 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
[Bug #14600]
Reported by MSP-Greg. Thanks!!!
* lib/rexml/xpath_parser.rb: Fix a bug that "following_siblings::*[N]"
doesn't work.
* test/rexml/test_jaxen.rb: Enable only axis test for now.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63057 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* lib/webrick/httpresponse.rb (send_body_io): use seek if NotImplementedError
is raised in IO.copy_stream with offset.
* lib/webrick/httpservlet/filehandler.rb (multipart_body): ditto.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63008 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Original patch by tenderlove (with minor style adjustments).
* lib/webrick/httpresponse.rb (send_header): call check_header
(check_header): raise on embedded CRLF in header value
* test/webrick/test_httpresponse.rb
(test_prevent_response_splitting_headers): new test
* (test_prevent_response_splitting_cookie_headers): ditto
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62968 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Reading entire request or response bodies into memory can lead
to trivial denial-of-service attacks. Introduce Fibers in both
cases to allow streaming.
WEBrick::HTTPRequest gains a new body_reader method to prepare
itself as a source for IO.copy_stream. This allows the
WEBrick::HTTPRequest object to be used as the
Net::HTTPGenericRequest#body_stream= arg for Net::HTTP.
For HTTP proxy response bodies, we also use a Fiber to
to make the HTTP request and read the response body.
* lib/webrick/httprequest.rb (body_reader): new method
(readpartial): ditto
* lib/webrick/httpproxy.rb (perform_proxy_request): use Fiber
to stream response body
(do_GET, do_HEAD): adjust call
(do_POST): adjust call and supply body_reader
* test/webrick/test_httprequest.rb (test_chunked): test
for IO.copy_stream compatibility
* test/webrick/test_httpproxy.rb (test_big_bodies): new test
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62966 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
WARNING! WARNING! WARNING! LIKELY BROKEN CHANGE
Pass a proc to WEBrick::HTTPRequest#body to avoid reading a
potentially large request body into memory during
authentication.
WARNING! this will break apps completely which want to do
something with the body besides calculating the MD5 digest
of it.
Also, keep in mind that probably nobody uses "auth-int".
Servers such as Apache, lighttpd, nginx don't seem to
support it; nor does curl when using POST/PUT bodies;
and we didn't have tests for it until now...
* lib/webrick/httpauth/digestauth.rb (_authenticate): stream req.body
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62965 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
While WEBrick::HTTPRequest#body provides a Proc interface
for streaming large request bodies, clients must not force
the server to use an excessively large chunk size.
* lib/webrick/httprequest.rb (read_chunk_size): limit each
read and block.call to :InputBufferSize in config.
* test/webrick/test_httpserver.rb (test_big_chunks): new test
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62963 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
"BadRequest" alone does not resolve correctly, it is in the
HTTPStatus namespace.
* lib/webrick/httprequest.rb (read_chunked): use correct exception
* test/webrick/test_httpserver.rb (test_eof_in_chunk): new test
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62962 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
WEBrick::HTTPRequest#body can be passed a block to process the
body in chunks. Use this feature to avoid building a giant
string in memory.
* lib/webrick/httpservlet/cgihandler.rb (do_GET):
avoid reading entire request body into memory
(do_POST is aliased to do_GET, so it handles bodies)
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62961 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
We use the same 112 KB limit started (AFAIK) by Mongrel, Thin,
and Puma to prevent malicious users from using up all the memory
with a single request. This also limits the damage done by
excessive ranges in multipart Range: requests.
Due to the way we rely on IO#gets and the desire to keep
the code simple, the actual maximum header may be 4093 bytes
larger than 112 KB, but we're splitting hairs at that point.
* lib/webrick/httprequest.rb: define MAX_HEADER_LENGTH
(read_header): raise when headers exceed max length
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62960 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Use the new Proc response body feature to generate a multipart
range response dynamically. We use a flat array to minimize
object overhead as much as possible; as many ranges may fit
into an HTTP request header.
* lib/webrick/httpservlet/filehandler.rb (multipart_body): new method
(make_partial_content): use multipart_body
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62959 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
The .write method needs to return the number of bytes written
to avoid confusing IO.copy_stream.
* lib/webrick/httpresponse.rb (ChunkedWrapper#write): return bytes written
(ChunkedWrapper#<<): return self
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62958 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Content-Length is ignored by WEBrick::HTTPResponse even if we
calculate it, so instead we chunk responses to HTTP/1.1 clients
and terminate HTTP/1.0 connections.
* lib/webrick/httpservlet/filehandler.rb (make_partial_content):
quiet warning
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62957 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
This is also compatible with range responses generated
by Rack::File (tested with rack 2.0.3).
* lib/webrick/httpresponse.rb (send_body_io): use Content-Range
* lib/webrick/httpservlet/filehandler.rb (make_partial_content):
use File object for the single range case
* test/webrick/test_filehandler.rb (get_res_body): use send_body
to test result
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62955 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Remove the redundant _send_file method since its functionality
is unnecessary with IO.copy_stream. IO.copy_stream also allows
the use of sendfile under some OSes to speed up copies to
non-TLS sockets.
Testing with "curl >/dev/null" and "ruby -run -e httpd" to
read a 1G file over Linux loopback reveals a reduction from
around ~0.770 to ~0.490 seconds on the client side.
* lib/webrick/httpresponse.rb (send_body_io): use IO.copy_stream
(_send_file): remove
[Feature #14237]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62954 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
This will make the next change to use IO.copy_stream
easier-to-read. When we can drop Ruby 2.4 support in a few
years, this will allow us to use writev(2) with multiple
arguments for headers and chunked responses.
* lib/webrick/cgi.rb (write): new wrapper method
lib/webrick/httpresponse.rb: (send_header): use socket.write
(send_body_io): ditto
(send_body_string): ditto
(send_body_proc): ditto
(_write_data): ditto
(ChunkedWrapper#write): ditto
(_send_file): ditto
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62953 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* lib/rdoc/text.rb: should escape `-` character.
Sometimes test fails if `$VERBOSE = 1` with the following warning:
> text.rb:172: warning: character class has duplicated range: ...
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62935 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
It fixed the several bugs that was found after RDoc 6 releasing.
From: SHIBATA Hiroshi <hsbt@ruby-lang.org>
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62924 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Now URI is normally UTF-8, and US-ASCII URI string is considered as
escaped a UTF-8 string.
https://github.com/rails/rails/issues/32294
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62897 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Improve code coverage and clarify meaning of hierarchical based on RFC
text.
[Fix GH-1846]
From: Xavier Riley <xavriley@hotmail.com>
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62882 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
If it will not use special variables (like $1, $&, $`...),
it can improve the performance by using Regexp#match? or String#match? instead of Regexp#=~ or String#=~.
This patch is same idea as https://github.com/ruby/ruby/pull/1836
[Fix GH-1842]
## Environment
* OS : Ubuntu 17.10
* Compiler : gcc version 7.2.0
* CPU : Intel(R) Core(TM) i5-3210M CPU @ 2.50GHz
* Memory : 16 GB
## TL;DR
Methods | Before | After | Speed up
----------- | ------ | ------ | --------
CSV.foreach | 44.825 | 48.201 | 7.5%
CSV#shift | 45.200 | 49.584 | 9.7%
CSV.read | 42.968 | 46.853 | 9.0%
CSV.table | 10.933 | 11.277 | 3.1%
## Before
```
Calculating -------------------------------------
CSV.foreach 44.825 (± 0.0%) i/s - 228.000 in 5.086576s
CSV#shift 45.200 (± 0.0%) i/s - 228.000 in 5.044297s
CSV.read 42.968 (± 0.0%) i/s - 216.000 in 5.027504s
CSV.table 10.933 (± 0.0%) i/s - 55.000 in 5.031098s
```
## After
```
Calculating -------------------------------------
CSV.foreach 48.201 (± 0.0%) i/s - 244.000 in 5.062256s
CSV#shift 49.584 (± 0.0%) i/s - 248.000 in 5.001652s
CSV.read 46.853 (± 0.0%) i/s - 236.000 in 5.037044s
CSV.table 11.277 (± 0.0%) i/s - 57.000 in 5.054694s
```
## Benchmark code
```ruby
require 'csv'
require 'benchmark/ips'
CSV.open("/tmp/file.csv", "w") do |csv|
csv << ["player", "gameA", "gameB"]
1000.times do
csv << ['"Alice"', "84.0", "79.5"]
csv << ['"Bob"', "20.0", "56.5"]
end
end
Benchmark.ips do |x|
x.report "CSV.foreach" do
CSV.foreach("/tmp/file.csv") do |row|
end
end
x.report "CSV#shift" do
CSV.open("/tmp/file.csv") do |csv|
while line = csv.shift
end
end
end
x.report "CSV.read" do
CSV.read("/tmp/file.csv")
end
x.report "CSV.table" do
CSV.table("/tmp/file.csv")
end
end
```
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62806 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* the default value of URI::File's authority is "" (localhost).
Both nil and "localhost" is normalized to "" by default.
* URI::File ignores setting userinfo and port
[Feature #14035]
fix https://github.com/ruby/ruby/pull/1719
fic https://github.com/ruby/ruby/pull/1832
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62767 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* lib/fileutils.rb: Add FileUtils#cp_lr. This method creates hard links
of each file from directory to another directory recursively.
This patch is based on Thomas Sawyers and Zachary Scott.
[Feature #4189] [ruby-core:33820]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62739 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
And thus avoiding Module#name calls. Those are slow, especially in
larger projects, with lots of anonymous modules.
[Fix GH-1798]
From: Dmitry Gutov <dgutov@yandex.ru>
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62732 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Reported by kivikakk (Ashe Connor) with tests and doc fix
Patch based on mame and fix by naruse
[Bug #14586]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62695 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* lib/resolv.rb (UnconnectedUDP#lazy_initialize): store new
sockets before binding, so the sockets get closed when the
requester is closing.
* lib/resolv.rb (ConnectedUDP#lazy_initialize): ditto.
* lib/resolv.rb (UnconnectedUDP#close): synchronize to get rid of
race condition.
* lib/resolv.rb (ConnectedUDP#close): ditto.
[ruby-core:85901] [Bug #14571]
From: quixoten (Devin Christensen) <quixoten@gmail.com>
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62671 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
I changed my mind and thought branching ERB.new in all libraries is too
hard. Code becomes too ugly. I increased the warn level to 2, and the
old initializer will be removed when Ruby 2.5 becomes EOL.
-S option of erb(1) stays in the same policy: will be removed at Ruby 2.7.
NEWS: note about the direction
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62612 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* In Enumerable, Enumerator::Lazy, Array, Hash and Set
[Feature #13784] [ruby-core:82285]
* Share specs for the various #select#select! methods and
reuse them for #filter/#filter!.
* Add corresponding filter tests for select tests.
* Update NEWS.
[Fix GH-1824]
From: Alexander Patrick <adp90@case.edu>
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62575 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Also, as it's in the middle of the list of 4 arguments, 3rd and 4th arguments
(trim_mode, eoutvar) are changed to keyword arguments.
Old ways to specify arguments are deprecated and warned now.
bin/erb: deprecate -S option.
We'll remove all of deprecated ones at Ruby 2.7+.
enc/make_encmake.rb: stopped using deprecated interface
ext/etc/mkconstants.rb: ditto
ext/socket/mkconstants.rb: ditto
sample/ripper/ruby2html.rb: ditto
spec/ruby/library/erb/defmethod/def_erb_method_spec.rb: ditto
spec/ruby/library/erb/new_spec.rb: ditto
test/erb/test_erb.rb: ditto
test/erb/test_erb_command.rb: ditto
tool/generic_erb.rb: ditto
tool/ruby_vm/helpers/dumper.rb: ditto
tool/transcode-tblgen.rb: ditto
lib/rdoc/erbio.rb: ditto
lib/rdoc/generator/darkfish.rb: ditto
[Feature #14256]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62529 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Patch by Yuuji Yaginuma. [Fixes GH-1808]
In other classes, `FrozenError` will be raised if change the frozen
object.
In order to match the behavior, I think that `FrozenError` should
use in `OpenStruct`.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62271 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Thanks to Paul Kuruvilla <rohitpaulk@gmail.com> for the patch
* lib/net/http.rb: fix documentation for HTTP connection reuse
[ruby-core:84815] [Bug #14349]
From: Paul Kuruvilla <rohitpaulk@gmail.com>
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62113 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Thanks to Michael Zimmerman for the bug report
* lib/net/pop.rb: make modified strings mutable
[ruby-core:85210] [Bug #14416]
* test/net/pop/test_pop.rb: new test
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62099 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Thanks to Sam Saffron for this patch, it shows
a nice reduction which affects many web applications:
require 'memory_profiler'
MemoryProfiler.report do
require 'uri'
end.pretty_print
Before:
Total allocated: 986643 bytes (15159 objects)
Total retained: 246370 bytes (2532 objects)
After:
Total allocated: 926903 bytes (13665 objects)
Total retained: 208570 bytes (1587 objects)
* lib/uri/common.rb: reduce allocations and retained objects
[ruby-core:85161] [Feature #14410]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62081 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* lib/optparse.rb (OptionParser#summarize): use literal newline to
join option summaries as IO#puts does, not the special gloval
variable $/.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62035 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* lib/optparse.rb (OptionParser#summarize): use literal newline to
join option summaries as IO#puts does, not the special gloval
variable $/.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62032 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* lib/resolv.rb (Resolv::Hosts::DefaultFileName): fallback to NUL
device when Win32::Resolv.get_hosts_path() returned nil, to get
rid of TypeError in lazy_initialize.
[ruby-core:84907] [Bug #14369] [Fix GH-1791]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61898 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* lib/resolv.rb (each_address): use safe navigation operator to
avoid extra hash lookups, as well as each_name since r56890.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61897 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Saves a few hundred bytes of bytecode in a frequently loaded module.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61894 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
This reduces both user and system CPU time for large
uploads with dynamically-generated request bodies.
user system total real
before: 0.393334 1.580000 1.973334 ( 1.971066)
after: 0.223334 0.976666 1.200000 ( 1.198514)
------
require 'socket'
require 'net/http'
require 'benchmark'
nr = 1024 * 1024 * 1024
s = TCPServer.new('127.0.0.1', 0)
addr = s.addr
at_exit { Process.waitall }
fork do
c = s.accept
# not exactly accurate but fast
IO.copy_stream(c, '/dev/null', nr + 500000)
begin
buf = c.readpartial(16384)
tmp = ''
until buf.end_with?(-"0\r\n\r\n")
buf << c.readpartial(16384, tmp)
end
rescue EOFError
end
c.write "HTTP/1.1 201 Created\r\nConnection:close\r\n\r\n"
c.close
end
r, w = IO.pipe
fork do
r.close
IO.copy_stream('/dev/zero', w, nr)
w.close
end
w.close
Net::HTTP.start(addr[3], addr[1]) do |http|
put = Net::HTTP::Put.new('/dev0/foo')
put['Content-Type'] = 'application/content-type'
put['Transfer-Encoding'] = 'chunked'
put.body_stream = r
puts(Benchmark.measure { http.request(put) })
end
------
* lib/net/http/generic_request.rb (write): use multi-arg write
* lib/net/protocol.rb (write): support multi-arg
(write0): ditto
[ruby-core:84845] [Feature #14339]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61812 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
We no longer need the compressed data once the inflate block is
called; so clear it ASAP to reduce memory overhead. This is a
small chunk, so it only saves a few hundred kilobytes with the
script below.
before: RssAnon: 5976 kB
after: RssAnon: 5564 kB
------
require 'net/http'
require 'zlib'
response_gz = ARGV.shift or abort "#$0 TEMPORARY_FILE"
# pre-create response since compressing is slower than decompressing
unless File.readable?(response_gz)
nr = 16384 * 2
buf = ((0..255).map(&:chr).join * 128)
File.open(response_gz, 'wb') do |fp|
gzip = Zlib::GzipWriter.new(fp)
nr.times { gzip.write(buf) }
gzip.close
end
buf.clear
end
response_gz = File.open(response_gz)
s = TCPServer.new('127.0.0.1', 0)
pid = fork do
c = s.accept
c.readpartial(16384).clear
c.write("HTTP/1.1 200 OK\r\n" \
"Content-Length: #{response_gz.stat.size}\r\n" \
"Content-Encoding: gzip\r\n" \
"Accept-Ranges: bytes\r\n" \
"\r\n")
IO.copy_stream(response_gz, c)
c.close
end
addr = s.addr
Net::HTTP.start(addr[3], addr[1]) do |http|
http.request_get(-'/') do |res|
res.read_body(&:clear)
end
end
puts File.readlines(-'/proc/self/status').grep(/RssAnon/)[0]
Process.waitpid2(pid)
------
* lib/net/http/response.rb (inflate_adapter): clear compressed_chunk
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61684 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Since r58846 (in Ruby 2.5), it is safe to clear the string
yielded to Net::HTTPResponse#read_body methods. This
reduces malloc garbage (anonymous RSS) using the Linux-only
script below:
before: user system total real
0.030000 0.250000 0.280000 ( 0.280511)
RssAnon: 60240 kB
after: user system total real
0.050000 0.223333 0.273333 ( 0.273118)
RssAnon: 6676 kB
------
# warning this script requires 1G free space for buffering
require 'open-uri'
require 'socket'
require 'benchmark'
s = TCPServer.new('127.0.0.1', 0)
len = 1024 * 1024 * 1024
buf = ((0..255).map(&:chr).join * 128)
nr = len / buf.size
pid = fork do
c = s.accept
c.readpartial(16384).clear
c.write("HTTP/1.1 200 OK\r\n" \
"Content-Length: #{len}\r\n" \
"Content-Type: application/octet-stream\r\n" \
"\r\n")
buf.freeze # speeds up IO#write slightly
nr.times { c.write(buf) }
c.close
end
addr = s.addr
open("http://#{addr[3]}:#{addr[1]}/", "rb") do |fp|
bm = Benchmark.measure do
while fp.read(16384, buf)
end
end
puts bm
end
puts File.readlines("/proc/#$$/status").grep(/RssAnon/)[0]
Process.waitpid2(pid)
------
* lib/open-uri.rb: clear string yielded by Net::HTTPResponse#read_body
[ruby-core:84662] [Feature #14320]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61664 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
There's no need to allocate a temporary string when @rbuf is
empty, we can use it as the read_nonblock destination buffer to
save both allocation overhead and avoid a later memcpy.
This results in a halving user CPU time and tiny memory
reduction with the script below:
user system total real
before 0.603333 0.539999 1.143332 ( 1.143347)
RssAnon: 5624 kB
after 0.283334 0.560000 0.843334 ( 0.846072)
RssAnon: 5592 kB
------
require 'net/http'
require 'benchmark'
s = TCPServer.new('127.0.0.1', 0)
len = 1024 * 1024 * 1024 * 2
pid = fork do
c = s.accept
c.readpartial(16384).clear
c.send("HTTP/1.0 200 OK\r\nContent-Length: #{len}\r\n\r\n", Socket::MSG_MORE)
IO.copy_stream('/dev/zero', c, len)
c.close
end
addr = s.addr
Net::HTTP.start(addr[3], addr[1]) do |http|
http.request_get('/') do |res|
puts(Benchmark.measure { res.read_body(&:clear) })
end
end
puts File.readlines("/proc/self/status").grep(/RssAnon/)[0]
Process.waitpid2(pid)
------
* lib/net/protocol.rb (rbuf_fill): avoid allocation if rbuf is empty
[ruby-core:84678] [Feature #14326]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61663 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Not an encoding expert, but this seems necessary for the next
change. All of the IO#read, IO#read_nonblock and related
methods will return a binary string when given a length
argument; so anything appended to these buffers via <<
will be binary.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61662 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Oops, not ready, yet (will work on this tomorrow :x).
This reverts commit r61638
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61640 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
I noticed this bug while working on something else with
RUBYOPT=-d on, existing test cases all passed with it.
Note: I use String.new because it is the local style, here,
I prefer +'' (or ''.b, for a future commit)
* lib/net/ftp.rb (BufferedSocket#read): use String.new
* test/net/ftp/test_buffered_socket.rb (test_read_nil): new test
[ruby-core:84675] [Bug #14323]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61639 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Not an encoding expert, but this seems necessary for the next
change.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61638 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
There are several places where rbuf_consume is called with
@rbuf.size as its length arg; simplify that case by avoiding
the slow String#slice! operation in favor of a lightweight
replacement.
The following script exhibits reduced memory usage and
runtimes using the time(1) command:
2.9s => 2.6s
70MB => 12 MB
---------
require 'net/http'
require 'digest/md5'
Thread.abort_on_exception = true
s = TCPServer.new('127.0.0.1', 0)
len = 1024 * 1024 * 1024
th = Thread.new do
c = s.accept
c.readpartial(16384)
c.write("HTTP/1.0 200 OK\r\nContent-Length: #{len}\r\n\r\n")
IO.copy_stream('/dev/zero', c, len)
c.close
end
addr = s.addr
Net::HTTP.start(addr[3], addr[1]) do |http|
http.request_get('/') do |res|
dig = Digest::MD5.new
res.read_body { |buf|
dig.update(buf)
# String#clear is important to reduce malloc overhead,
# but most Ruby programmers don't do this :<
buf.clear
}
puts dig.hexdigest
end
end
----------
* lib/net/protocol (rbuf_consume): optimize for @rbuf.size == len
[Feature #14268]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61602 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* lib/rdoc/test_case.rb (setup): call `RDoc::Markup::PreProcess.reset`
at the end of `setup` method.
`RDoc::RDoc.new` requires `rdoc/generator/darkfish`
and requires `rdoc/tom_doc.rb` at last. It add post_proecssor
(at `add_post_processor`) and it fails
`TestRDocMarkupPreProcess#test_class_post_process`.
This issue occured only with sorted tests by alphabetical order.
`make test-all TESTS='rdoc/test_rdoc_markup_pre_process.rb --test-order=alpha`)
We can observe this failure with parallel test only a few times.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61521 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* vm_core.h (rb_vm_t): move `rb_execution_context_t::safe_level` to
`rb_vm_t::safe_level_` because `$SAFE` is a process (VM) global state.
* vm_core.h (rb_proc_t): remove `rb_proc_t::safe_level` because `Proc`
objects don't need to keep `$SAFE` at the creation.
Also make `is_from_method` and `is_lambda` as 1 bit fields.
* cont.c (cont_restore_thread): no need to keep `$SAFE` for Continuation.
* eval.c (ruby_cleanup): use `rb_set_safe_level_force()` instead of access
`vm->safe_level_` directly.
* eval_jump.c: End procs `END{}` doesn't keep `$SAFE`.
* proc.c (proc_dup): removed and introduce `rb_proc_dup` in vm.c.
* safe.c (rb_set_safe_level): don't check `$SAFE` 1 -> 0 changes.
* safe.c (safe_setter): use `rb_set_safe_level()`.
* thread.c (rb_thread_safe_level): `Thread#safe_level` returns `$SAFE`.
It should be obsolete.
* transcode.c (load_transcoder_entry): `rb_safe_level()` only returns
0 or 1 so that this check is not needed.
* vm.c (vm_proc_create_from_captured): don't need to keep `$SAFE` for Proc.
* vm.c (rb_proc_create): renamed to `proc_create`.
* vm.c (rb_proc_dup): moved from proc.c.
* vm.c (vm_invoke_proc): do not need to set and restore `$SAFE`
for `Proc#call`.
* vm_eval.c (rb_eval_cmd): rename a local variable to represent clearer
meaning.
* lib/drb/drb.rb: restore `$SAFE`.
* lib/erb.rb: restore `$SAFE`, too.
* test/lib/leakchecker.rb: check `$SAFE == 0` at the end of tests.
* test/rubygems/test_gem.rb: do not set `$SAFE = 1`.
* bootstraptest/test_proc.rb: catch up this change.
* spec/ruby/optional/capi/string_spec.rb: ditto.
* test/bigdecimal/test_bigdecimal.rb: ditto.
* test/fiddle/test_func.rb: ditto.
* test/fiddle/test_handle.rb: ditto.
* test/net/imap/test_imap_response_parser.rb: ditto.
* test/pathname/test_pathname.rb: ditto.
* test/readline/test_readline.rb: ditto.
* test/ruby/test_file.rb: ditto.
* test/ruby/test_optimization.rb: ditto.
* test/ruby/test_proc.rb: ditto.
* test/ruby/test_require.rb: ditto.
* test/ruby/test_thread.rb: ditto.
* test/rubygems/test_gem_specification.rb: ditto.
* test/test_tempfile.rb: ditto.
* test/test_tmpdir.rb: ditto.
* test/win32ole/test_win32ole.rb: ditto.
* test/win32ole/test_win32ole_event.rb: ditto.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61510 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
comment. Fix regression at r58948.
I even don't want to deprecate it because deprecation needs to lex all
embedded Ruby script using Ripper and it would be slow. So Let me just
keep this behavior of Ruby 2.4. No change is the best compatibility.
This commit stopped using String#-@ because it's harmful for "ambiguous
first argument" warning if we really want to maintain this behavior.
[Bug #14243]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61497 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* lib/rubygems/command.rb (Gem::Command#invoke_with_build_args):
restore UI and close the created SilentUI.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61465 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
This release removes uses of Kernel#open to avoid unintended
behaviors and make future auditing easier. [Misc #14216]
6 changes since 1.4.1:
webrick: httpauth requires regular files
webrick/httpservlet/cgi_runner.rb: remove unnecessary open
webrick: WEBrick::Log requires path arg when given string
webrick/httpservlet/*handler: use File.open
webrick/httputils: note Kernel#open behavior
webrick/httpservelet/cgi_runner: avoid IO#reopen on pathname
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61443 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
It fixes an installation error of activerecord.
https://github.com/ruby/rdoc/issues/571
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61432 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
I faced a big issue about Bundler with ruby core.
I have no time to resolve it issue before 2.5 final release.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61416 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* lib/erb.rb (ERB#new_toplevel): shadow already defined local
variables by block local variabes, not to overwrite them.
[ruby-core:84390] [Bug #14215]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61411 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
IO#reopen seems to behave strangely on Win32 based on the logs
posted by Greg for [Bug #14420]. Lets try to fix the issue by
using File.open explicitly (but not Kernel#open).
* lib/webrick/httpservelet/cgi_runner: use File.open explicitly
[Bug #14220]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61409 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
I don't know who uses the load_mime_types method; but it is
conceivable that a user would want to read the results of a
command instead of reading a regular file to load MIME types.
None of the WEBrick-related code in Ruby or default/bundled gems
seems to rely on this method; but it is likely 3rd-party code does.
* lib/webrick/httputils.rb (load_mime_types): note Kernel#open behavior
[Misc #14216]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61402 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
This makes future code audits easier. None of these changes
fix realistic remote code execution vulnerabilities because
we stat(2) before attempting Kernel#open.
* lib/webrick/httpservlet/erbhandler.rb (do_GET): use File.open
* lib/webrick/httpservlet/filehandler.rb (do_GET): use File.open
(make_partial_content): ditto
[Misc #14216]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61401 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Allowing a user to specify "| command" via Kernel#open is
nonsensical since we never read from the resultant IO.
* lib/webrick/log.rb (initialize): replace Kernel#open with File.open
[Misc #14216]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61400 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
IO#reopen already takes string path names as well as IO objects
(but not "| command" strings)
This makes further auditing for inadvertant code execution
easier. There's no actual bugfix or behavior change here,
as no external data is passed to cgi_runner.rb.
* lib/webrick/httpservlet/cgi_runner.rb: remove Kernel#open call
[Misc #14216]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61398 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Be sure we do not try to open a pipe to read from, since we care
about mtime in all cases.
* lib/webrick/httpauth/htdigest.rb: use File.open
* lib/webrick/httpauth/htgroup.rb: ditto
* lib/webrick/httpauth/htpasswd.rb: ditto
[Misc #14216]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61397 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
like r59693, initialize_http_header also should raise error. [Bug #14208]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61359 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* lib/webrick/version.rb: change VERSION to 1.4.0
* lib/webrick/webrick.gemspec: require_relative for version,
update date and contact info
[Feature #13173]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61266 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* generic.rb (URI::Generic#merge, URI::Generic#route_to): Fix a bug
where a sequence of slashes in the path part gets collapsed to a
single slash. According to the relevant RFCs and WHATWG URL
Standard, empty path components are simply valid and there is no
special treatment defined for them, so we just keep them as they
are.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61218 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Using ^ and $ in regexps means we can accidentally get fooled
by "%0a" in HTTP request paths being decoded to newline
characters. Use \A and \z to match beginning and end-of-string
respectively, instead.
Thanks to mame and hsbt for reporting.
* lib/webrick/httpserver.rb (MountTable#compile):
use \A and \z instead of ^ and $
* lib/webrick/httpserver.rb (MountTable#normalize): use \z instead of $
* test/webrick/test_httpserver.rb (test_cntrl_in_path): new test
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61197 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* lib/drb/drb.rb: avoid creating a Thread and call the shutdown logic
directly. Do not try to kill or join the current Thread.
Thread.new { stop_service } caused "can't alloc thread (ThreadError)",
which is shown with Thread.report_on_exception = true.
[Bug #14171]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61184 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
If uplevel keyword is given, the warning message is prepended
with caller file and line information and the string "warning: ".
The use of the uplevel keyword makes Kernel#warn format output
similar to how rb_warn formats output.
This patch modifies net/ftp and net/imap to use Kernel#warn
instead of $stderr.puts or $stderr.printf, since they are used
for printing warnings.
This makes lib/cgi/core and tempfile use $stderr.puts instead of
warn for debug logging, since they are used for debug printing
and not for warning.
This does not modify bundler, rubygems, or rdoc, as those are
maintained outside of ruby and probably wish to remain backwards
compatible with older ruby versions.
rb_warn_m code is originally from nobu, but I've changed it
so that it only includes the path and lineno from uplevel
(not the method), and also prepends the string "warning: ",
to make it more similar to rb_warn.
From: Jeremy Evans code@jeremyevans.net
Signed-off-by: Urabe Shyouhei shyouhei@ruby-lang.org
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61155 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
This is not perfectly good solution (at least we don't want to have ARGV
as default value of `argv` argument), but unfortunately IRB.setup and
IRB.parse_opts are public methods and we can't make breaking change to
those methods.
We may deprecate using them and then make them private in the future,
but the removal should not be in Ruby 2.5. So I kept their interface for
now.
[Bug #14162] [close GH-1770]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61149 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
It's rc version for bundler-1.16.1. I'm going to update it version
after official release from bundler team.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61134 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Because there is now the same guard in prelude.rb (alias pp pp).
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61111 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* lib/mkmf.rb (depend_rules): all objs should depend on win32.h on Windows.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61107 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
We must to ensure the @status ivar is set to :Running before
running StartCallback, otherwise Webrick::Server#stop will not
change the @status to :Shutdown properly.
Note: I have not been able to reproduce the original issue but
understood at least part of the problem and fixed it with this
commit. However, the original reporter (Peak Xu) was still able
to reproduce the problem on 1.9.2 p180 on Windows, so I'm not
sure what else might be going on. Ruby threading and
synchronization primitives have changed a lot since 1.9.2, so
maybe that was fixed elsewhere.
* lib/webrick/server.rb: call StartCallback sooner [Bug #4841]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61083 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* lib/pp.rb (pp): move pp alias before its rdoc, not to prevent
parsing.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61080 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
https://url.spec.whatwg.org/#url-serializing
> Otherwise, if url’s host is null and url’s scheme is "file", append "//" to output.
URL spec doesn't says anything about postgres, but assume the same thing.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@60970 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
nobu
kou
mame
ko1
naruse
usa
hsbt
normal
svn
watson1978
seki
kazu
eregon
k0kubun
marcandre
stomar
shyouhei
sorah
akr
sonots
nagachika
knu