Граф коммитов

87 Коммитов

Автор SHA1 Сообщение Дата
Martin Dürst c06f9e1d24 switch UNICODE_BETA to NO (one more try, first try didn't work)
Unicode version 12.1.0 was officially released on May 7th, 2019.
    There were no changes at all from the "real" beta
    published shortly after the new era name "Reiwa" was announced.
    So we can switch UNICODE_BETA back to NO.

    common.mk: switch UNICODE_BETA back to NO
    tool/downloader.rb: add additional conditions to avoid an error
                        when moving from beta to final
2019-05-24 18:22:36 +09:00
Kazuhiro NISHIYAMA 0eedec6867
Change Accept-Encoding from `*` to `identity`
When `Accept-Encoding` is `*`,
http://www.unicode.org/Public/12.1.0/ucd/ returns gzipped content now.
So set `identity`.
2019-04-30 22:35:38 +09:00
Takashi Kokubun 3067370f61
Retry downloads more for unicode outage
like https://ci.appveyor.com/project/ruby/ruby/builds/24142523/job/v6aq4srj7c3hgt86
2019-04-27 22:21:36 +09:00
nobu ffbcf9f842 downloader.rb: enable cache files
* tool/downloader.rb (Downloader::Unicode.download): enable cache
  files when index.html has not been modified since the previous
  download, even if beta version.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67594 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-04-17 07:35:29 +00:00
nobu 5141398ff0 downloader.rb: use File.read instead of IO.read
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67591 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-04-17 07:09:28 +00:00
nobu b5df5296a4 appveyor.yml: Use pre-generated headers and tables
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67454 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-04-06 00:08:04 +00:00
duerst cab005db92 revert r67445, (r67446,) r67447
Debugging output is no longer needed because the problem has been fixed with r67449.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67451 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-04-05 23:16:00 +00:00
nobu 6e0e93e60e downloader.rb: fix typo, extra %
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67450 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-04-05 14:44:15 +00:00
nobu aebc11dfe6 downloader.rb: quote base name
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67448 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-04-05 12:44:04 +00:00
duerst 8f734dc7b5 output more debug information in downloader.rb (temporary)
Unicode file download doesn't work with Visual Studio, we need more debug output.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67447 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-04-05 12:36:29 +00:00
svn f6338966f1 * remove trailing spaces.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67446 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-04-05 10:39:02 +00:00
duerst 7fd8cdf398 add puts statements to debug Unicode file download (temporary)
Unicode file download doesn't work with Visual Studio, see e.g.
https://ci.appveyor.com/project/ruby/ruby/builds/23614399/job/f8vya2l7fjdfcye4
We temporarily produce more output for debugging.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67445 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-04-05 10:39:01 +00:00
nobu 79de8e9f6b downloader.rb: keep linked file newer than cached file
* tool/downloader.rb (Downloader.save_cache): keep linked file
  newer than cached file, so that GNU make triggers when the
  content is updated.  it uses the timestamp of symlink itself
  instead of the target.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67413 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-04-02 08:08:29 +00:00
duerst d9699fadf8 simplify filename-related code
(Thanks to Nobuyoshi Nakada for the hint!)

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66738 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-01-06 11:41:12 +00:00
duerst b991ccc210 add logic to handle Unicode beta period file names
In downloader.rb, add logic to handle file names of the form
UnicodeData-12.0.0d6.txt. To find the right file, we download
the index of the directory. Then we download the files by finding
the file names from the index. Files are always checked for changes,
because changes might be frequent during the beta period.
We also check whether any index.html files are left when we are not
in the beta period. This would indicate that we might have stale
data from the beta period rather than the actual release data.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66737 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-01-06 07:50:18 +00:00
duerst 13c15beb68 avoid passing unnecessary options to download method
The option --unicode-beta for tool/downloader.rb introduced in
r66448 must not be passed as an option to actual download machinery.
Thanks to MSP-Greg (Greg L.) for bug report and patch.
This closes issue #15434.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66449 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-12-19 04:08:46 +00:00
duerst eb2e931f6f add option to deal with Unicode beta data files
Unicode uses file names with explicit versions for beta publication.
This commit introduces a variable on the makefile level to distinguish
between beta and regular versions of file names.
 
common.mk: Define new variable UNICODE_BETA, usually set to NO,
but would be YES during tests with beta data files. Pass the value
of this variable to tool/downloader.rb with option --unicode-beta.

tool/downloader.rb: Receive and store value of --unicode-beta.
Raise an exception if value is YES, because we don't yet actually
deal with this case. Continue as usual if value is not YES.

This completes the changes needed in the makefile. Upcomming changes
will only affect tool/downloader.rb.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66448 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-12-19 00:26:31 +00:00
k0kubun 57005046a1 tool/downloader.rb: retry Net::OpenTimeout
to prevent random failure
https://travis-ci.org/ruby/ruby/jobs/461667474

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66110 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-12-01 00:37:54 +00:00
k0kubun c312111c7c tool/downloader.rb: retry Net::ReadTimeout
as failed in
https://ci.appveyor.com/project/ruby/ruby/builds/19540849/job/9p9pq0b7hfrb2r8q

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65097 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-10-16 09:29:23 +00:00
k0kubun 288e27899b tool/downloader.rb: retry 500 from GitHub
Unfortunately, GitHub may return 500 for temporary failure:
https://travis-ci.org/ruby/ruby/builds/417246523

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64425 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-08-18 02:04:44 +00:00
k0kubun 4fe8308e0c tool/downloader.rb: increase retries
GitHub download failed on 13:50:36
https://ci.appveyor.com/project/ruby/ruby/build/1.0.9221 and it also
failed on 13:51:35 (all builds between them failed too).

It means that we need to expect GitHub 502 that continues 1 minute.
So I configured 6 retries, that will sleep at most 91s in total.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64317 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-08-12 06:00:31 +00:00
k0kubun 92c03be888 tool/downloader.rb: retry on 502 and 503 error
rubyci was failed by download 503
https://rubyci.org/logs/rubyci.s3.amazonaws.com/scw-9d6766/ruby-trunk/log/20180811T021706Z.fail.html.gz

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64282 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-08-11 04:22:14 +00:00
duerst abb7438849 tool/downloader.rb:
* remove initial "./" from destdir to make downloading
  ./enc/unicode/data/11.0.0/ucd/auxiliary/GraphemeBreakProperty.txt
  work properly
* tweak a comment

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63978 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-07-16 10:20:24 +00:00
k0kubun 349e7fcb66 tool/downloader.rb: retry SocketError
I would like to stabilize RubyCI.

https://rubyci.org/logs/rubyci.s3.amazonaws.com/fedora26/ruby-trunk/log/20180205T123003Z.fail.html.gz

downloading did_you_mean-1.2.0.gem ... /home/hsbt/chkbuild/tmp/build/20180205T123003Z/tmp/ruby-snapshot20180205-23226-3201ha/ruby-2.6.0-r62226/tool/downloader.rb:212:in `rescue in download': failed to download did_you_mean-1.2.0.gem (RuntimeError)
SocketError: Failed to open TCP connection to rubygems.org:443 (getaddrinfo: Name or service not known): https://rubygems.org/downloads/did_you_mean-1.2.0.gem
	from /home/hsbt/chkbuild/tmp/build/20180205T123003Z/tmp/ruby-snapshot20180205-23226-3201ha/ruby-2.6.0-r62226/tool/downloader.rb:123:in `download'
	from /home/hsbt/chkbuild/tmp/build/20180205T123003Z/tmp/ruby-snapshot20180205-23226-3201ha/ruby-2.6.0-r62226/tool/downloader.rb:66:in `download'
	from -e:4:in `<main>'
make[1]: *** [Makefile:1685: update-gems] Error 1

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62231 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-02-05 13:24:00 +00:00
k0kubun 35e631273a tool/downloader.rb: retry against Errno::ETIMEDOUT
instead of Errno::ECONNREFUSED.

As I commented in r61498, at that moment I was not sure if the actual
exception is really `Errno::ECONNREFUSED` or not.

In https://ci.appveyor.com/project/ruby/ruby/build/1.0.6974, I could
confirm that the download can fail with `Errno::ETIMEDOUT`. So I want to
retry that.

Let's add `Errno::ECONNREFUSED` too if it fails with the exception.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61762 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-01-10 15:44:52 +00:00
k0kubun 57e95f38b1 tool/downloader.rb: don't suppress exception
on the end of retry failure.

Fixed mistake in r61498

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61499 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-12-27 13:35:25 +00:00
k0kubun ff6c540adb tool/downloader.rb: retry downloads
because it's randomly failing on CI like
https://ci.appveyor.com/project/ruby/ruby/build/1.0.6724

Actually I'm not sure whether the exception class is Errno::ECONNREFUSED
or not. Please change the rescued exception to the correct one if it's
wrong. I changed to log exception class too in this commit.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61498 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-12-27 13:32:59 +00:00
hsbt 774f650bbe Revert "switch from http to https for Unicode data file downloads"
This reverts commit 90de118a75.

  The environment of mswinci couldn't handle https download.
  (It's certificates problem, We should upgrade its environment)

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61169 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-12-12 14:10:43 +00:00
duerst 90de118a75 switch from http to https for Unicode data file downloads
(patch from MSP-Greg (Greg L), this closes issue #13962)

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61145 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-12-12 09:00:20 +00:00
nobu b391708718 downloader.rb: get rid of symlinks in dist files
* tool/downloader.rb: download to the target file directly if
  CACHE_SAVE=no, to get rid of symlinks in dist files.
  [ruby-core:81887] [Bug #13709]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@59292 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-07-09 04:49:15 +00:00
nobu 5522d79803 downloader.rb: get rid of symlinks
* tool/downloader.rb: do not save downloaded files to cache.
  [ruby-core:81780] [Bug #13684]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@59185 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-06-27 17:10:56 +00:00
nobu c45a52af36 downloader.rb: true symlink on cygwin
* tool/downloader.rb (Downloader.link_cache): on Cygwin, make true
  symlink only, which is provided by the OS.  as mingw/mswin ruby
  can't follow cygwin's pseudo symlink, it will fail on the same
  source tree.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58843 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-05-22 04:10:51 +00:00
nobu 7a14ffbf5d fix r58833
* tool/downloader.rb (Downloader.download): just link to the cache
  when downloaded to the cache but not the target file.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58841 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-05-22 03:18:17 +00:00
nobu 48070cef94 downloader cache
* tool/downloader.rb (Downloader.download): manage download cache.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58833 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-05-21 16:45:35 +00:00
nobu 2ee57d5ff1 remove verify option
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58832 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-05-21 14:29:00 +00:00
nobu 70b9957689 downloader.rb: utilize dir argument
* tool/downloader.rb (Downloader::RubyGems.download): utilize
  `dir` argument of Downloader.download.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58831 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-05-21 08:30:50 +00:00
nobu 81ab413288 fix GraphemeBreakProperty.txt
* tool/downloader.rb: download to the file given in ARGV.

* tool/enc-unicode.rb (parse_GraphemeBreakProperty): fix data file
  path as $(UNICODE_PROPERTY_FILES) in common.mk.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58069 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-03-23 15:49:10 +00:00
nobu 3805c4442b tool/downloader.rb: dryrun option [ci skip]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58043 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-03-21 07:19:23 +00:00
naruse c11e648799 Regexp supports Unicoe 9.0.0's \X
* meta character \X matches Unicode 9.0.0 characters with some workarounds
  for UTR #51 Unicode Emoji, Version 4.0 emoji zwj sequences.
  [Feature #12831] [ruby-core:77586]

The term "character" can have many meanings bytes, codepoints, combined
characters, and so on. "grapheme cluster" is highest one of such words,
which means user-perceived characters.
Unicode Standard Annex #29 UNICODE TEXT SEGMENTATION specifies how to
handle grapheme clusters (extended grapheme cluster).
But some specs aren't updated to current situation because Unicode Emoji
is rapidly extended without well definition.
It breaks the precondition of UTR#29 "Grapheme cluster boundaries can be
easily tested by looking at immediately adjacent characters". (the
sentence will be removed in the next version)
Though some of its detail are described in Unicode Technical Report #51
UNICODE EMOJI but it is not merged into UTR#29 yet.

http://unicode.org/reports/tr29/
http://unicode.org/reports/tr51/
http://unicode.org/Public/emoji/4.0/

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@56949 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-11-30 17:29:19 +00:00
hsbt 2e5fffa7cf * tool/downloader.rb: Removed verification of gem certification.
Because signed gem is not working on rubygems ecosystem.
* tool/gem-unpack.rb: ditto.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@56399 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-10-12 06:24:59 +00:00
normal f8d0bdedf1 tool: add descriptions and fix typos
* tool/asm_parse.rb: add description
* tool/change_maker.rb: ditto
* tool/downloader.rb: ditto
* tool/eval.rb: ditto
* tool/expand-config.rb: ditto
* tool/extlibs.rb: ditto
* tool/fake.rb: ditto
* tool/file2lastrev.rb: ditto
* tool/gem-unpack.rb: ditto
* tool/gen_dummy_probes.rb: ditto
* tool/gen_ruby_tapset.rb: ditto
* tool/generic_erb.rb: ditto
* tool/id2token.rb: ditto
* tool/ifchange: ditto
* tool/insns2vm.rb: ditto
* tool/instruction.rb: ditto
* tool/jisx0208.rb: ditto
* tool/merger.rb: ditto
* tool/mkrunnable.rb: ditto
* tool/node_name.rb: ditto
* tool/parse.rb: ditto
* tool/rbinstall.rb: ditto
* tool/rbuninstall.rb: ditto
* tool/rmdirs: ditto
* tool/runruby.rb: ditto
* tool/strip-rdoc.rb: ditto
* tool/vcs.rb: ditto
* tool/vtlh.rb: ditto
* tool/ytab.sed: ditto
* tool/enc-unicode.rb: fix typo
* tool/mk_call_iseq_optimized.rb: ditto
* tool/update-deps: ditto
  [ruby-core:76215] [Bug #12539]
  by Noah Gibbs <the.codefolio.guy@gmail.com>

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@55564 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-07-02 21:01:04 +00:00
nobu 37beb43b55 downloader.rb: disable verify if rubygems is old
* tool/downloader.rb (Downloader::RubyGems.download): verify gems
  only if RubyGems is 2.4 or later.  old RubyGems fails to verify
  almost all of bundled gems.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@55055 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-05-18 07:04:55 +00:00
usa 3f1fc5e42b * tool/downloader.rb (RubyGems.download): follow the change of the
rubygems ssl_certs directory tree introduced by previous commit.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@54502 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-04-06 08:40:02 +00:00
nobu 38c0bdccdb downloader.rb: defer warning
* tool/downloader.rb (Downloader.download): warn using http only
  when really downloading.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53865 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-02-18 07:53:13 +00:00
nobu df64c27ef8 downloader.rb: https setter
* tool/downloader.rb (Downloader.https): define class variable
  setter for 1.8, class_variable_set was private.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53717 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-02-02 07:08:13 +00:00
duerst 39988ff0ac * tool/downloader.rb: Fixed a logical error, improved documentation
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53631 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-01-23 07:30:32 +00:00
nobu f78ab819bf common.mk: get rid of repeating same names
* common.mk (download-unicode-data): strip directory paths from
  the target names, to get rid of repeating same file names.

* tool/downloader.rb: add -p option for prefix.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53574 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-01-18 10:21:03 +00:00
naruse 8f0268991c Use more tough server: GitHub or repo.or.cz
https://gcc.gnu.org/wiki/GitMirror

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@52761 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2015-11-26 15:40:58 +00:00
nobu a3ddead991 downloader.rb: Downloader.https
* tool/downloader.rb (Downloader.https): extract a method instead
  of a global variable.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@52758 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2015-11-26 09:09:49 +00:00
kazu 894507c5f4 fix a typo [ci skip]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@50175 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2015-04-06 15:00:00 +00:00