We no longer need the compressed data once the inflate block is
called; so clear it ASAP to reduce memory overhead. This is a
small chunk, so it only saves a few hundred kilobytes with the
script below.
before: RssAnon: 5976 kB
after: RssAnon: 5564 kB
------
require 'net/http'
require 'zlib'
response_gz = ARGV.shift or abort "#$0 TEMPORARY_FILE"
# pre-create response since compressing is slower than decompressing
unless File.readable?(response_gz)
nr = 16384 * 2
buf = ((0..255).map(&:chr).join * 128)
File.open(response_gz, 'wb') do |fp|
gzip = Zlib::GzipWriter.new(fp)
nr.times { gzip.write(buf) }
gzip.close
end
buf.clear
end
response_gz = File.open(response_gz)
s = TCPServer.new('127.0.0.1', 0)
pid = fork do
c = s.accept
c.readpartial(16384).clear
c.write("HTTP/1.1 200 OK\r\n" \
"Content-Length: #{response_gz.stat.size}\r\n" \
"Content-Encoding: gzip\r\n" \
"Accept-Ranges: bytes\r\n" \
"\r\n")
IO.copy_stream(response_gz, c)
c.close
end
addr = s.addr
Net::HTTP.start(addr[3], addr[1]) do |http|
http.request_get(-'/') do |res|
res.read_body(&:clear)
end
end
puts File.readlines(-'/proc/self/status').grep(/RssAnon/)[0]
Process.waitpid2(pid)
------
* lib/net/http/response.rb (inflate_adapter): clear compressed_chunk
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61684 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
There's no need to allocate a temporary string when @rbuf is
empty, we can use it as the read_nonblock destination buffer to
save both allocation overhead and avoid a later memcpy.
This results in a halving user CPU time and tiny memory
reduction with the script below:
user system total real
before 0.603333 0.539999 1.143332 ( 1.143347)
RssAnon: 5624 kB
after 0.283334 0.560000 0.843334 ( 0.846072)
RssAnon: 5592 kB
------
require 'net/http'
require 'benchmark'
s = TCPServer.new('127.0.0.1', 0)
len = 1024 * 1024 * 1024 * 2
pid = fork do
c = s.accept
c.readpartial(16384).clear
c.send("HTTP/1.0 200 OK\r\nContent-Length: #{len}\r\n\r\n", Socket::MSG_MORE)
IO.copy_stream('/dev/zero', c, len)
c.close
end
addr = s.addr
Net::HTTP.start(addr[3], addr[1]) do |http|
http.request_get('/') do |res|
puts(Benchmark.measure { res.read_body(&:clear) })
end
end
puts File.readlines("/proc/self/status").grep(/RssAnon/)[0]
Process.waitpid2(pid)
------
* lib/net/protocol.rb (rbuf_fill): avoid allocation if rbuf is empty
[ruby-core:84678] [Feature #14326]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61663 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Not an encoding expert, but this seems necessary for the next
change. All of the IO#read, IO#read_nonblock and related
methods will return a binary string when given a length
argument; so anything appended to these buffers via <<
will be binary.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61662 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Oops, not ready, yet (will work on this tomorrow :x).
This reverts commit r61638
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61640 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
I noticed this bug while working on something else with
RUBYOPT=-d on, existing test cases all passed with it.
Note: I use String.new because it is the local style, here,
I prefer +'' (or ''.b, for a future commit)
* lib/net/ftp.rb (BufferedSocket#read): use String.new
* test/net/ftp/test_buffered_socket.rb (test_read_nil): new test
[ruby-core:84675] [Bug #14323]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61639 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Not an encoding expert, but this seems necessary for the next
change.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61638 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
There are several places where rbuf_consume is called with
@rbuf.size as its length arg; simplify that case by avoiding
the slow String#slice! operation in favor of a lightweight
replacement.
The following script exhibits reduced memory usage and
runtimes using the time(1) command:
2.9s => 2.6s
70MB => 12 MB
---------
require 'net/http'
require 'digest/md5'
Thread.abort_on_exception = true
s = TCPServer.new('127.0.0.1', 0)
len = 1024 * 1024 * 1024
th = Thread.new do
c = s.accept
c.readpartial(16384)
c.write("HTTP/1.0 200 OK\r\nContent-Length: #{len}\r\n\r\n")
IO.copy_stream('/dev/zero', c, len)
c.close
end
addr = s.addr
Net::HTTP.start(addr[3], addr[1]) do |http|
http.request_get('/') do |res|
dig = Digest::MD5.new
res.read_body { |buf|
dig.update(buf)
# String#clear is important to reduce malloc overhead,
# but most Ruby programmers don't do this :<
buf.clear
}
puts dig.hexdigest
end
end
----------
* lib/net/protocol (rbuf_consume): optimize for @rbuf.size == len
[Feature #14268]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61602 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
like r59693, initialize_http_header also should raise error. [Bug #14208]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61359 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
If uplevel keyword is given, the warning message is prepended
with caller file and line information and the string "warning: ".
The use of the uplevel keyword makes Kernel#warn format output
similar to how rb_warn formats output.
This patch modifies net/ftp and net/imap to use Kernel#warn
instead of $stderr.puts or $stderr.printf, since they are used
for printing warnings.
This makes lib/cgi/core and tempfile use $stderr.puts instead of
warn for debug logging, since they are used for debug printing
and not for warning.
This does not modify bundler, rubygems, or rdoc, as those are
maintained outside of ruby and probably wish to remain backwards
compatible with older ruby versions.
rb_warn_m code is originally from nobu, but I've changed it
so that it only includes the path and lineno from uplevel
(not the method), and also prepends the string "warning: ",
to make it more similar to rb_warn.
From: Jeremy Evans code@jeremyevans.net
Signed-off-by: Urabe Shyouhei shyouhei@ruby-lang.org
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61155 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
require_relative speeds up loading of files by reducing path
lookups. On a clean install with RubyGems-enabled,
"ruby -rnet/http -e exit" shows a reduction in failed open(2)
syscalls from 410 to 350 (x86-64 GNU/Linux).
I could not measure a time difference on my Linux-based
machines, however this should be noticeable to users of other
kernels with worse syscall and VFS performance than Linux.
Further use of require_relative will reduce lookups in other
places.
* lib/net/http.rb: use require_relative
[ruby-core:78285] [Feature #12973]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@60388 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* lib/net/ftp.rb (Net::FTP#initialize): simplify as per
the original intent.
* spec/ruby/library/net/ftp/initialize_spec.rb: adapt specs.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@59981 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* lib/net/http/header.rb (set_field): `val` can not have `to_str`.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@59695 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
The IMAP server of DOCOMO returns such continuation requests.
[ruby-list:50558]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@59666 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Based on the patch by keysen (Jérémy Carlier).
[ruby-core:81641] [Bug #13649]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@59372 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Users may modify the chunk yielded to them in Net::HTTPResponse#read_body.
This will allow users to reduce memory usage by calling
String#clear on the buffer once they're done using it.
* lib/net/protocol.rb (read): increment read_bytes earlier
(read_all): ditto
* test/net/http/test_httpresponse.rb (test_read_body_block_mod): new test
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58846 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Otherwise literal data will be sent even if NO response is returned
because @exception is set to nil in receive_responses.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58844 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Using a parallel Net::HTTP downloader, this reduced memory usage
from around 120MB to 50MB on my 32-bit x86 system.
* lib/net/protocol.rb (rbuf_fill): clear temporary buffer
Test script I used:
require 'net/http'
require 'uri'
require 'digest/sha1'
url = 'http://80x24.org/git-i-forgot-to-pack/objects/pack/pack-97b25a76c03b489d4cbbd85b12d0e1ad28717e55.idx'
uri = URI(url)
use_ssl = "https" == uri.scheme
thrs = 30.times.map do
Thread.start do
cur = Thread.current.object_id
Net::HTTP.start(uri.host, uri.port, use_ssl: use_ssl) do |http|
req = Net::HTTP::Get.new(uri)
http.request(req) do |res|
dig = Digest::SHA1.new
res.read_body do |buf|
dig.update(buf)
#buf.clear # most Ruby programmers don't do this :<
end
warn "#{Time.now} #{cur} #{dig.hexdigest}\n"
end
end
:done
end
end
p thrs.map(&:value)
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58840 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
get_response is called in a receiver thread, so there may be no pending
commands when get_response is called.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58680 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Patch by Pavel Rosický. [Feature #13379] [ruby-core:80440]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58549 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Note that this feature is enabled only on environment variables are
multi-user safe. In this time the list includes Linux, FreeBSD, or
Darwin. [Bug #12921]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58461 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
BufferedIO#rbuf_fill should preserve backtrace information when raising
EOFError. Otherwise, users get confused when EOFError is leaked out from
Net::SMTP etc. [ruby-core:78550] [Bug #13018]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@57311 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
The TLS handshake timeout can be specified independently, because the TLS
handshake doesn't start just after the underlying connection is established,
but after the AUTH command is completed. It's also useful for testing
purposes. However, if ssl_handshake_timeout is not specified, open_timeout
is used instead for convenience.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@56919 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
In that case, BufferedSSLSocket#read in FTP#close exceeded timeout because
BufferedSSLSocket#shutdown did nothing. So BufferedIO#rbuf_fill is
overridden in BufferedSSLSocket to raise an EOFError if the connection is
shut down.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@56880 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Based on the patch by soleboxy.
[Fix GH-1478] [ruby-core:78240] [Feature #12965]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@56879 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
SSLSocket#send is not defined, so use #write instead. flags and dest are
ignored.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@56871 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* lib/net/ftp.rb (Net::FTP#initialize): [DOC] fix type in option
name, :username is used but :use is not.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@56862 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
When net/http connects to an HTTPS server through a CONNECT proxy, it
writes the CONNECT request to an unconnected OpenSSL::SSL::SSLSocket.
OpenSSL::SSL::SSLSocket traditionally fallbacks to a method call on the
underlying IO object if a read/write method is called before the TLS
connection is established. So it automagically works correctly, emitting
the "SSL session is not started yet" warning.
This is not obvious at first glance. The warning is also noisy. Let's
just write to the plain socket instead of relying on the SSLSocket's
behavior.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@56857 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Because @sock.remote_address fails if @sock is an SSLSocket.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@56847 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
SSLSocket#stop is a private method and cannot be called, but explicit calls
are not necessary because SSL_shutdown() is called from SSLSocket#close.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@56846 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
If the :ssl options is specified, the control connection is protected with
TLS in the manner described in RFC 4217. Data connections are also
protected with TLS unless the :private_data_connection is set to false.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@56834 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
in 1xx (HTTPInformation) also needs to continue. [Bug #12890]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@56596 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* lib/net/http.rb (Net::HTTP#proxy_uri): cache the case no proxy
is used.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@55520 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* lib/net/http.rb (Net::HTTP#proxy): use conditional operator to
get rid of confusing ruby-mode.el.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@55519 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* lib/net/smtp.rb: [DOC] Remove dead link on documentation for
Japanese of SMTP. [Fix GH-1380]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@55379 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
CR or LF is included in a line, because they are not allowed in
RFC5321.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@55324 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Net::HTTPHeader#{each_header,each_name,each_capitalized_name,
each_value,each_capitalized} without block returns an enumerator.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@54907 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
kazu
normal
naruse
nagachika
shyouhei
stomar
hsbt
eregon
ko1
shugo
svn
nobu
rhe
a_matsuda
ktsj