This commit should fix Net::Protocol::BufferedIO#write when sending
large multi-byte string like following example.
```
$ ruby -rnet/http -rjson -v -e "Net::HTTP.post(URI('http://httpbin.org/post'), { text: 'あ'*100_000 }.to_json, 'Content-Type' => 'application/json')"
ruby 2.6.0p0 (2018-12-25 revision 66547) [x86_64-linux]
Traceback (most recent call last):
19: from -e:1:in `<main>'
18: from lib/ruby/2.6.0/net/http.rb:500:in `post'
17: from lib/ruby/2.6.0/net/http.rb:605:in `start'
16: from lib/ruby/2.6.0/net/http.rb:920:in `start'
15: from lib/ruby/2.6.0/net/http.rb:502:in `block in post'
14: from lib/ruby/2.6.0/net/http.rb:1281:in `post'
13: from lib/ruby/2.6.0/net/http.rb:1493:in `send_entity'
12: from lib/ruby/2.6.0/net/http.rb:1479:in `request'
11: from lib/ruby/2.6.0/net/http.rb:1506:in `transport_request'
10: from lib/ruby/2.6.0/net/http.rb:1506:in `catch'
9: from lib/ruby/2.6.0/net/http.rb:1507:in `block in transport_request'
8: from lib/ruby/2.6.0/net/http/generic_request.rb:123:in `exec'
7: from lib/ruby/2.6.0/net/http/generic_request.rb:189:in `send_request_with_body'
6: from lib/ruby/2.6.0/net/protocol.rb:247:in `write'
5: from lib/ruby/2.6.0/net/protocol.rb:265:in `writing'
4: from lib/ruby/2.6.0/net/protocol.rb:248:in `block in write'
3: from lib/ruby/2.6.0/net/protocol.rb:275:in `write0'
2: from lib/ruby/2.6.0/net/protocol.rb:275:in `each_with_index'
1: from lib/ruby/2.6.0/net/protocol.rb:275:in `each'
lib/ruby/2.6.0/net/protocol.rb:280:in `block in write0': undefined method `bytesize' for nil:NilClass (NoMethodError)
```
[Fix GH-2058]
From: Eito Katagiri <eitoball@gmail.com>
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66582 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* lib/net/protocol.rb (ReadTimeout, WriteTimeout): Net::ReadTimeout and Net::WriteTimeout should tell the cause socket
[Feature #14832] [ruby-core:87440]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66308 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
This reduces both user and system CPU time for large
uploads with dynamically-generated request bodies.
user system total real
before: 0.393334 1.580000 1.973334 ( 1.971066)
after: 0.223334 0.976666 1.200000 ( 1.198514)
------
require 'socket'
require 'net/http'
require 'benchmark'
nr = 1024 * 1024 * 1024
s = TCPServer.new('127.0.0.1', 0)
addr = s.addr
at_exit { Process.waitall }
fork do
c = s.accept
# not exactly accurate but fast
IO.copy_stream(c, '/dev/null', nr + 500000)
begin
buf = c.readpartial(16384)
tmp = ''
until buf.end_with?(-"0\r\n\r\n")
buf << c.readpartial(16384, tmp)
end
rescue EOFError
end
c.write "HTTP/1.1 201 Created\r\nConnection:close\r\n\r\n"
c.close
end
r, w = IO.pipe
fork do
r.close
IO.copy_stream('/dev/zero', w, nr)
w.close
end
w.close
Net::HTTP.start(addr[3], addr[1]) do |http|
put = Net::HTTP::Put.new('/dev0/foo')
put['Content-Type'] = 'application/content-type'
put['Transfer-Encoding'] = 'chunked'
put.body_stream = r
puts(Benchmark.measure { http.request(put) })
end
------
* lib/net/http/generic_request.rb (write): use multi-arg write
* lib/net/protocol.rb (write): support multi-arg
(write0): ditto
[ruby-core:84845] [Feature #14339]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61812 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
There's no need to allocate a temporary string when @rbuf is
empty, we can use it as the read_nonblock destination buffer to
save both allocation overhead and avoid a later memcpy.
This results in a halving user CPU time and tiny memory
reduction with the script below:
user system total real
before 0.603333 0.539999 1.143332 ( 1.143347)
RssAnon: 5624 kB
after 0.283334 0.560000 0.843334 ( 0.846072)
RssAnon: 5592 kB
------
require 'net/http'
require 'benchmark'
s = TCPServer.new('127.0.0.1', 0)
len = 1024 * 1024 * 1024 * 2
pid = fork do
c = s.accept
c.readpartial(16384).clear
c.send("HTTP/1.0 200 OK\r\nContent-Length: #{len}\r\n\r\n", Socket::MSG_MORE)
IO.copy_stream('/dev/zero', c, len)
c.close
end
addr = s.addr
Net::HTTP.start(addr[3], addr[1]) do |http|
http.request_get('/') do |res|
puts(Benchmark.measure { res.read_body(&:clear) })
end
end
puts File.readlines("/proc/self/status").grep(/RssAnon/)[0]
Process.waitpid2(pid)
------
* lib/net/protocol.rb (rbuf_fill): avoid allocation if rbuf is empty
[ruby-core:84678] [Feature #14326]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61663 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Not an encoding expert, but this seems necessary for the next
change. All of the IO#read, IO#read_nonblock and related
methods will return a binary string when given a length
argument; so anything appended to these buffers via <<
will be binary.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61662 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Oops, not ready, yet (will work on this tomorrow :x).
This reverts commit r61638
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61640 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Not an encoding expert, but this seems necessary for the next
change.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61638 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
There are several places where rbuf_consume is called with
@rbuf.size as its length arg; simplify that case by avoiding
the slow String#slice! operation in favor of a lightweight
replacement.
The following script exhibits reduced memory usage and
runtimes using the time(1) command:
2.9s => 2.6s
70MB => 12 MB
---------
require 'net/http'
require 'digest/md5'
Thread.abort_on_exception = true
s = TCPServer.new('127.0.0.1', 0)
len = 1024 * 1024 * 1024
th = Thread.new do
c = s.accept
c.readpartial(16384)
c.write("HTTP/1.0 200 OK\r\nContent-Length: #{len}\r\n\r\n")
IO.copy_stream('/dev/zero', c, len)
c.close
end
addr = s.addr
Net::HTTP.start(addr[3], addr[1]) do |http|
http.request_get('/') do |res|
dig = Digest::MD5.new
res.read_body { |buf|
dig.update(buf)
# String#clear is important to reduce malloc overhead,
# but most Ruby programmers don't do this :<
buf.clear
}
puts dig.hexdigest
end
end
----------
* lib/net/protocol (rbuf_consume): optimize for @rbuf.size == len
[Feature #14268]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61602 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Users may modify the chunk yielded to them in Net::HTTPResponse#read_body.
This will allow users to reduce memory usage by calling
String#clear on the buffer once they're done using it.
* lib/net/protocol.rb (read): increment read_bytes earlier
(read_all): ditto
* test/net/http/test_httpresponse.rb (test_read_body_block_mod): new test
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58846 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Using a parallel Net::HTTP downloader, this reduced memory usage
from around 120MB to 50MB on my 32-bit x86 system.
* lib/net/protocol.rb (rbuf_fill): clear temporary buffer
Test script I used:
require 'net/http'
require 'uri'
require 'digest/sha1'
url = 'http://80x24.org/git-i-forgot-to-pack/objects/pack/pack-97b25a76c03b489d4cbbd85b12d0e1ad28717e55.idx'
uri = URI(url)
use_ssl = "https" == uri.scheme
thrs = 30.times.map do
Thread.start do
cur = Thread.current.object_id
Net::HTTP.start(uri.host, uri.port, use_ssl: use_ssl) do |http|
req = Net::HTTP::Get.new(uri)
http.request(req) do |res|
dig = Digest::SHA1.new
res.read_body do |buf|
dig.update(buf)
#buf.clear # most Ruby programmers don't do this :<
end
warn "#{Time.now} #{cur} #{dig.hexdigest}\n"
end
end
:done
end
end
p thrs.map(&:value)
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58840 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
BufferedIO#rbuf_fill should preserve backtrace information when raising
EOFError. Otherwise, users get confused when EOFError is leaked out from
Net::SMTP etc. [ruby-core:78550] [Bug #13018]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@57311 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
When you change this to true, you may need to add more tests.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53141 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
io/wait is expected to work on any platform where sockets are
supported. io/wait methods uses fewer allocations and uses
ppoll internally under Linux for better performance on
high-numbered FDs.
[ruby-core:35572] describes the performance advantage of ppoll
on high-numbered FDs.
* lib/net/protocol.rb (rbuf_fill): use IO#wait_*able
* lib/net/http/generic_request.rb (wait_for_continue): ditto
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@50326 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Exceptions are noisy in debug output and waste allocations.
Use "exception: false" introduced in 2.1 to return symbols for
common errors instead.
Follow-up commits will be prepared to reduce EOFError exceptions
to further quiet debug output and IO.select may be replaced by
io/wait methods if available to reduce allocations.
[ruby-core:68787] [Feature #11044]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@50219 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
OpenTimeout. ReadTimeout is now raised by rbuf_fill instead of
Timeout::Error to help users distinguish what type of timeout
occurred. [ruby-trunk - Feature #6088]
* lib/net/pop.rb (module Net): Updated documentation for ReadTimeout
and OpenTimeout.
* lib/net/http.rb (module Net): ditto
* lib/net/smtp.rb (module Net): ditto
* lib/net/telnet.rb (module Net): Net::ReadTimeout is now raised in
waitfor to match Net::Protocol.
* test/net/http/test_http.rb: Updated Timeout::Error expectation to
Net::ReadTimeout.
* test/net/ftp/test_ftp.rb: ditto
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@35304 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
server returning '100 continue' response befor sending HTTP request
body. See NEWS for more detail. See #3622.
Original patch is made by Eric Hodel <drbrain@segment7.net>.
* test/net/http/test_http.rb: test it.
* NEWS: Add new feature.
On my env (Ubuntu 11.04 64bit),
9510 tests, 2203824 assertions, 0 failures, 0 errors, 29 skips
->
9514 tests, 2203836 assertions, 0 failures, 0 errors, 29 skips
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@31860 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
both application/x-www-form-urlencoded and multipart/form-data.
There is a similar API, Net::HTTPRequest#set_form_data, but
to keep its compatibility this is newly added. [ruby-dev:42729]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@30188 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
read_nonblock instead of sysread wrapped by timeout to boost
performance. a patch from Aaron Patterson in [ruby-core:20191].
fix#806
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@20443 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* lib/net/smtp.rb: check server advertisement.
* lib/net/smtp.rb: introduce new class SMTP::Response.
* lib/net/smtp.rb (getok): should not use sprintf.
* lib/net/smtp.rb (get_response): ditto.
* lib/net/protocol.rb: reduce syntax warning on 1.9.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@11994 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Jeremy Evans
nobu
usa
kazu
naruse
ko1
normal
shugo
drbrain
nahi
tenderlove
akr
matz
aamine
gsinclair