Граф коммитов

102 Коммитов

Автор SHA1 Сообщение Дата
Alan Wu 83f9d80c0b
downloader.rb: unlink broken symlink before caching to fix ENOENT
On GitHub CI, the downloader's cache is shared across different jobs.
For some systems, the cached config.guess ends up being a symlink to
/usr/share/autoconf/build-aux/config.guess and containers that don't
have that file end up consuming the cache anyways, leading to ENOENT
when trying to cache the downloaded file.

This error happened on forks:
 - https://github.com/XrXr/ruby/actions/runs/5675262636/job/15380232344
 - https://github.com/peterzhu2118/ruby/actions/runs/5684765421/job/15408188728
2023-07-27 17:09:08 -04:00
Nobuyoshi Nakada c2671fa051
downloader.rb: Select less components path 2022-12-03 02:03:23 +09:00
Nobuyoshi Nakada e558c374b7
downloader.rb: Fix link to absolute cache path 2022-12-03 01:28:34 +09:00
Nobuyoshi Nakada cc533cb607
Downloader: Define long option aliases 2022-09-19 01:09:47 +09:00
Nobuyoshi Nakada a0b0991eed
Downloader: Define per-class command line options
Move `Downloader::Unicode` specific options, and parse options after
the downloader specificier.
2022-09-19 01:09:47 +09:00
Nobuyoshi Nakada e76e1d3ce4
Downloader: retry when RFC 2616 noncompliant dates [ci skip]
zlib.net rarely returns the current time in RFC 2616 noncompliant
format in the response header, and the checksum does not match in
that case (maybe creating the tarball on the fly?).
2021-10-28 17:42:36 +09:00
Nobuyoshi Nakada c8a1e57af0
Downloader: accept some RFC 2616 noncompliant dates 2021-09-13 01:48:43 +09:00
Takashi Kokubun 53e352fd71
Increase the # of downloader.rb's retry attempts
Apparently 9 was not enough either.
https://rubyci.org/logs/rubyci.s3.amazonaws.com/rhel_zlinux/ruby-master/log/20201123T063303Z.fail.html.gz
2020-11-23 00:06:51 -08:00
Kazuhiro NISHIYAMA 4b899f9164 Try to fix download error
https://github.com/ruby/ruby/runs/1428320660?check_suite_focus=true#step:9:10
```
tool/downloader.rb:243:in `rescue in download': failed to download config.guess (RuntimeError)
OpenURI::HTTPError: 403 Forbidden: https://cdn.jsdelivr.net/gh/gcc-mirror/gcc@master/config.guess
```
2020-11-20 14:18:14 +09:00
Hiroshi SHIBATA 544ebc669b
Use https for downloading Unicode data.
Retry with 90de118a75

  [Bug #13962][ruby-core:83074]
2020-03-09 13:39:23 +09:00
Kazuhiro NISHIYAMA 8eb0a9e566
Use jsDelivr instead of raw.githubusercontent.com
Try to fix download error on Solaris CI

https://rubyci.org/logs/rubyci.s3.amazonaws.com/unstable11x/ruby-master/log/20191121T162422Z.fail.html.gz
```
tool/downloader.rb:243:in `rescue in download': failed to download config.guess (RuntimeError)
Net::HTTPFatalError: 503 "Service Unavailable": https://raw.githubusercontent.com/gcc-mirror/gcc/master/config.guess
```
2019-11-22 13:55:59 +09:00
Jeremy Evans d2ba80b5df Revert "Fix building with 1.8 BASERUBY"
This reverts commit 05bc14d81a.

We have decided that the cost of reintroducing support for 1.8
BASERUBY outweighs the benefit.  If you are still using 1.8 and want
to build master/trunk, build and install the latest release, and use
that as BASERUBY.
2019-05-29 18:03:21 -07:00
Nobuyoshi Nakada 40e175b390
Clean a garbage [ci skip] 2019-05-28 14:20:59 +09:00
Jeremy Evans 05bc14d81a Fix building with 1.8 BASERUBY 2019-05-27 20:55:56 -07:00
Kazuhiro NISHIYAMA 706c816ad2
Escape dots in regexp 2019-05-24 19:03:29 +09:00
Martin Dürst c06f9e1d24 switch UNICODE_BETA to NO (one more try, first try didn't work)
Unicode version 12.1.0 was officially released on May 7th, 2019.
    There were no changes at all from the "real" beta
    published shortly after the new era name "Reiwa" was announced.
    So we can switch UNICODE_BETA back to NO.

    common.mk: switch UNICODE_BETA back to NO
    tool/downloader.rb: add additional conditions to avoid an error
                        when moving from beta to final
2019-05-24 18:22:36 +09:00
Kazuhiro NISHIYAMA 0eedec6867
Change Accept-Encoding from `*` to `identity`
When `Accept-Encoding` is `*`,
http://www.unicode.org/Public/12.1.0/ucd/ returns gzipped content now.
So set `identity`.
2019-04-30 22:35:38 +09:00
Takashi Kokubun 3067370f61
Retry downloads more for unicode outage
like https://ci.appveyor.com/project/ruby/ruby/builds/24142523/job/v6aq4srj7c3hgt86
2019-04-27 22:21:36 +09:00
nobu ffbcf9f842 downloader.rb: enable cache files
* tool/downloader.rb (Downloader::Unicode.download): enable cache
  files when index.html has not been modified since the previous
  download, even if beta version.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67594 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-04-17 07:35:29 +00:00
nobu 5141398ff0 downloader.rb: use File.read instead of IO.read
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67591 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-04-17 07:09:28 +00:00
nobu b5df5296a4 appveyor.yml: Use pre-generated headers and tables
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67454 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-04-06 00:08:04 +00:00
duerst cab005db92 revert r67445, (r67446,) r67447
Debugging output is no longer needed because the problem has been fixed with r67449.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67451 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-04-05 23:16:00 +00:00
nobu 6e0e93e60e downloader.rb: fix typo, extra %
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67450 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-04-05 14:44:15 +00:00
nobu aebc11dfe6 downloader.rb: quote base name
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67448 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-04-05 12:44:04 +00:00
duerst 8f734dc7b5 output more debug information in downloader.rb (temporary)
Unicode file download doesn't work with Visual Studio, we need more debug output.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67447 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-04-05 12:36:29 +00:00
svn f6338966f1 * remove trailing spaces.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67446 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-04-05 10:39:02 +00:00
duerst 7fd8cdf398 add puts statements to debug Unicode file download (temporary)
Unicode file download doesn't work with Visual Studio, see e.g.
https://ci.appveyor.com/project/ruby/ruby/builds/23614399/job/f8vya2l7fjdfcye4
We temporarily produce more output for debugging.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67445 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-04-05 10:39:01 +00:00
nobu 79de8e9f6b downloader.rb: keep linked file newer than cached file
* tool/downloader.rb (Downloader.save_cache): keep linked file
  newer than cached file, so that GNU make triggers when the
  content is updated.  it uses the timestamp of symlink itself
  instead of the target.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67413 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-04-02 08:08:29 +00:00
duerst d9699fadf8 simplify filename-related code
(Thanks to Nobuyoshi Nakada for the hint!)

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66738 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-01-06 11:41:12 +00:00
duerst b991ccc210 add logic to handle Unicode beta period file names
In downloader.rb, add logic to handle file names of the form
UnicodeData-12.0.0d6.txt. To find the right file, we download
the index of the directory. Then we download the files by finding
the file names from the index. Files are always checked for changes,
because changes might be frequent during the beta period.
We also check whether any index.html files are left when we are not
in the beta period. This would indicate that we might have stale
data from the beta period rather than the actual release data.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66737 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-01-06 07:50:18 +00:00
duerst 13c15beb68 avoid passing unnecessary options to download method
The option --unicode-beta for tool/downloader.rb introduced in
r66448 must not be passed as an option to actual download machinery.
Thanks to MSP-Greg (Greg L.) for bug report and patch.
This closes issue #15434.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66449 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-12-19 04:08:46 +00:00
duerst eb2e931f6f add option to deal with Unicode beta data files
Unicode uses file names with explicit versions for beta publication.
This commit introduces a variable on the makefile level to distinguish
between beta and regular versions of file names.
 
common.mk: Define new variable UNICODE_BETA, usually set to NO,
but would be YES during tests with beta data files. Pass the value
of this variable to tool/downloader.rb with option --unicode-beta.

tool/downloader.rb: Receive and store value of --unicode-beta.
Raise an exception if value is YES, because we don't yet actually
deal with this case. Continue as usual if value is not YES.

This completes the changes needed in the makefile. Upcomming changes
will only affect tool/downloader.rb.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66448 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-12-19 00:26:31 +00:00
k0kubun 57005046a1 tool/downloader.rb: retry Net::OpenTimeout
to prevent random failure
https://travis-ci.org/ruby/ruby/jobs/461667474

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66110 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-12-01 00:37:54 +00:00
k0kubun c312111c7c tool/downloader.rb: retry Net::ReadTimeout
as failed in
https://ci.appveyor.com/project/ruby/ruby/builds/19540849/job/9p9pq0b7hfrb2r8q

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65097 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-10-16 09:29:23 +00:00
k0kubun 288e27899b tool/downloader.rb: retry 500 from GitHub
Unfortunately, GitHub may return 500 for temporary failure:
https://travis-ci.org/ruby/ruby/builds/417246523

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64425 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-08-18 02:04:44 +00:00
k0kubun 4fe8308e0c tool/downloader.rb: increase retries
GitHub download failed on 13:50:36
https://ci.appveyor.com/project/ruby/ruby/build/1.0.9221 and it also
failed on 13:51:35 (all builds between them failed too).

It means that we need to expect GitHub 502 that continues 1 minute.
So I configured 6 retries, that will sleep at most 91s in total.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64317 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-08-12 06:00:31 +00:00
k0kubun 92c03be888 tool/downloader.rb: retry on 502 and 503 error
rubyci was failed by download 503
https://rubyci.org/logs/rubyci.s3.amazonaws.com/scw-9d6766/ruby-trunk/log/20180811T021706Z.fail.html.gz

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64282 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-08-11 04:22:14 +00:00
duerst abb7438849 tool/downloader.rb:
* remove initial "./" from destdir to make downloading
  ./enc/unicode/data/11.0.0/ucd/auxiliary/GraphemeBreakProperty.txt
  work properly
* tweak a comment

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63978 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-07-16 10:20:24 +00:00
k0kubun 349e7fcb66 tool/downloader.rb: retry SocketError
I would like to stabilize RubyCI.

https://rubyci.org/logs/rubyci.s3.amazonaws.com/fedora26/ruby-trunk/log/20180205T123003Z.fail.html.gz

downloading did_you_mean-1.2.0.gem ... /home/hsbt/chkbuild/tmp/build/20180205T123003Z/tmp/ruby-snapshot20180205-23226-3201ha/ruby-2.6.0-r62226/tool/downloader.rb:212:in `rescue in download': failed to download did_you_mean-1.2.0.gem (RuntimeError)
SocketError: Failed to open TCP connection to rubygems.org:443 (getaddrinfo: Name or service not known): https://rubygems.org/downloads/did_you_mean-1.2.0.gem
	from /home/hsbt/chkbuild/tmp/build/20180205T123003Z/tmp/ruby-snapshot20180205-23226-3201ha/ruby-2.6.0-r62226/tool/downloader.rb:123:in `download'
	from /home/hsbt/chkbuild/tmp/build/20180205T123003Z/tmp/ruby-snapshot20180205-23226-3201ha/ruby-2.6.0-r62226/tool/downloader.rb:66:in `download'
	from -e:4:in `<main>'
make[1]: *** [Makefile:1685: update-gems] Error 1

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62231 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-02-05 13:24:00 +00:00
k0kubun 35e631273a tool/downloader.rb: retry against Errno::ETIMEDOUT
instead of Errno::ECONNREFUSED.

As I commented in r61498, at that moment I was not sure if the actual
exception is really `Errno::ECONNREFUSED` or not.

In https://ci.appveyor.com/project/ruby/ruby/build/1.0.6974, I could
confirm that the download can fail with `Errno::ETIMEDOUT`. So I want to
retry that.

Let's add `Errno::ECONNREFUSED` too if it fails with the exception.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61762 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-01-10 15:44:52 +00:00
k0kubun 57e95f38b1 tool/downloader.rb: don't suppress exception
on the end of retry failure.

Fixed mistake in r61498

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61499 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-12-27 13:35:25 +00:00
k0kubun ff6c540adb tool/downloader.rb: retry downloads
because it's randomly failing on CI like
https://ci.appveyor.com/project/ruby/ruby/build/1.0.6724

Actually I'm not sure whether the exception class is Errno::ECONNREFUSED
or not. Please change the rescued exception to the correct one if it's
wrong. I changed to log exception class too in this commit.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61498 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-12-27 13:32:59 +00:00
hsbt 774f650bbe Revert "switch from http to https for Unicode data file downloads"
This reverts commit 90de118a75.

  The environment of mswinci couldn't handle https download.
  (It's certificates problem, We should upgrade its environment)

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61169 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-12-12 14:10:43 +00:00
duerst 90de118a75 switch from http to https for Unicode data file downloads
(patch from MSP-Greg (Greg L), this closes issue #13962)

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61145 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-12-12 09:00:20 +00:00
nobu b391708718 downloader.rb: get rid of symlinks in dist files
* tool/downloader.rb: download to the target file directly if
  CACHE_SAVE=no, to get rid of symlinks in dist files.
  [ruby-core:81887] [Bug #13709]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@59292 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-07-09 04:49:15 +00:00
nobu 5522d79803 downloader.rb: get rid of symlinks
* tool/downloader.rb: do not save downloaded files to cache.
  [ruby-core:81780] [Bug #13684]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@59185 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-06-27 17:10:56 +00:00
nobu c45a52af36 downloader.rb: true symlink on cygwin
* tool/downloader.rb (Downloader.link_cache): on Cygwin, make true
  symlink only, which is provided by the OS.  as mingw/mswin ruby
  can't follow cygwin's pseudo symlink, it will fail on the same
  source tree.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58843 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-05-22 04:10:51 +00:00
nobu 7a14ffbf5d fix r58833
* tool/downloader.rb (Downloader.download): just link to the cache
  when downloaded to the cache but not the target file.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58841 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-05-22 03:18:17 +00:00
nobu 48070cef94 downloader cache
* tool/downloader.rb (Downloader.download): manage download cache.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58833 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-05-21 16:45:35 +00:00
nobu 2ee57d5ff1 remove verify option
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58832 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-05-21 14:29:00 +00:00