Thanks for the patch gareth (Gareth Adams). [Bug #15933]
-------
Combines two small, but very related changes
1: Treat HTTPS the same as HTTP
Previously, OpenURI followed guidance in RFC2616/3.7.1:
> When no explicit charset parameter is provided by the sender, media
> subtypes of the "text" type are defined to have a default charset
> value of "ISO-8859-1" when received via HTTP.
However this RFC was written before TLS was established and OpenURI was
never updated to treat HTTPS traffic the same way. So, HTTPS documents
received a different default to HTTP documents.
This commit removes the scheme check so that all text/* documents
processed by OpenURI are treated the same way.
In theory this processing gets applied to FTP URIs too, but there's no
mechanism in OpenURI for FTP documents to have Content-Type metadata
appended to them, so this ends up being a no-op.
2: Change default charset for text/* to UTF-8
Replaces the default ISO-8859-1 charset previously defined in RFC2616 (now
obsoleted) with a UTF-8 charset as defined in RFC6838.
Fixes: https://bugs.ruby-lang.org/issues/15933
Since r58846 (in Ruby 2.5), it is safe to clear the string
yielded to Net::HTTPResponse#read_body methods. This
reduces malloc garbage (anonymous RSS) using the Linux-only
script below:
before: user system total real
0.030000 0.250000 0.280000 ( 0.280511)
RssAnon: 60240 kB
after: user system total real
0.050000 0.223333 0.273333 ( 0.273118)
RssAnon: 6676 kB
------
# warning this script requires 1G free space for buffering
require 'open-uri'
require 'socket'
require 'benchmark'
s = TCPServer.new('127.0.0.1', 0)
len = 1024 * 1024 * 1024
buf = ((0..255).map(&:chr).join * 128)
nr = len / buf.size
pid = fork do
c = s.accept
c.readpartial(16384).clear
c.write("HTTP/1.1 200 OK\r\n" \
"Content-Length: #{len}\r\n" \
"Content-Type: application/octet-stream\r\n" \
"\r\n")
buf.freeze # speeds up IO#write slightly
nr.times { c.write(buf) }
c.close
end
addr = s.addr
open("http://#{addr[3]}:#{addr[1]}/", "rb") do |fp|
bm = Benchmark.measure do
while fp.read(16384, buf)
end
end
puts bm
end
puts File.readlines("/proc/#$$/status").grep(/RssAnon/)[0]
Process.waitpid2(pid)
------
* lib/open-uri.rb: clear string yielded by Net::HTTPResponse#read_body
[ruby-core:84662] [Feature #14320]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61664 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* lib/open-uri.rb: Allow http to https redirection.
Note that https to http is still forbidden.
[ruby-core:20485] [Feature #859] by Roman Shterenzon.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@56085 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
default now. The default mode can be changed by
Net::FTP.default_passive=.
* lib/net/ftp.rb (default_passive=, default_passive): new methods.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@52532 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
ssl_ca_certs.
* tool/downloader.rb: use certs of rubygems for downloading gems.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48941 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
the StringIO object is not closed yet.
Reported by Jordi Massaguer Pla. [ruby-core:42538] [Bug #6010]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@45835 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
as commit r44878, based on patch by Jonathan Jackson [Bug #9483] [ci skip]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@44879 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
name (like Set-Cookie).
(OpenURI::Meta#metas): New accessor to obtain fields as a Hash from
field name (string) to field values (array of strings).
[ruby-core:37734] [Bug #4964] reported by ren li.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@40207 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
proxies from the environment. A proxy may also be specified as
before.
Net::HTTP::Proxy still creates anonymous classes, but these classes
are only used to store configuration information. When an HTTP
instance is created the configuration is now copied.
Additionally, Net::HTTP::ProxyDelta is no longer used by Net::HTTP
[Feature #6546]
* lib/open-uri.rb: Moved URI::Generic#find_proxy to uri/generic.
* lib/uri/generic.rb: Imported find_proxy from open-uri.
* test/open-uri/test_open-uri.rb: Moved proxy-discovery tests to URI.
* test/uri/test_generic.rb: Imported proxy-discovery tests from
open-uri.
* test/net/http/test_http.rb: Added tests for proxy behavior.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@36476 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
by charset.
(OpenURI::Meta#meta_add_field): call meta_setup_encoding when
content-type.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@15630 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
(OpenURI::HTTPRedirect): new exception class for redirection.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@13788 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Tanaka Akira
normal
akr
hsbt
shugo
usa
zzak
drbrain
ayumin
naruse
matz