OpenSSL's man page previously stated that "the application is
responsible for seeding the PRNG by calling RAND_add" (see [1]).
So we had this code. However things changed. They no longer
say so, instead "manual (re-)seeding of the default OpenSSL
random generator is not necessary" now (see [2]). It seems all
OpenSSL versions that we support now already behaves like this.
Let's follow that.
[1]: https://www.openssl.org/docs/man1.0.2/man3/RAND_add.html
[2]: https://www.openssl.org/docs/manmaster/man3/RAND_add.html
(https://github.com/ruby/irb/pull/559)
* Remove unnecessary command argument generation code
Now that all the supported Ruby versions handle splat args and splat kwargs,
we don't need that args generation code anymore.
* Remove unused command definition code
If we look at `@EXTEND_COMMANDS`, all commands are defined in a file, which
means the `if load_file` branch is always executed. Therefore we can drop
the else branch of that condition.
* Avoid defining unnecessary command methods
There's no need to define another command method just to call `Command.execute`.
If Gemfile has a lot of dependencies, we have an optimization that uses
the full index in that case, assuming it's going to be faster.
I think this is an old optimization that predates compact index API
times, I believe we no longer need it these days.
Also, since a few releases ago we check for circular dependencies when
resolving by looping through all versions of each name and removing
those that have circular dependencies that would trip up the resolver.
This loop becomes actually very slow when full indexes are used because
to find dependencies of a gemspec, we need to explicitly fetch the
marshaled gemspec (`gemspec.rz` endpoint) for it, so the optimization
has the opposite effect of making things very slow.
https://github.com/rubygems/rubygems/commit/2f46289bd3
Fix a CI error and add a test to ensure we're testing the current version:
```
Run bundle exec rake test
bundler: failed to load command: rake (/home/runner/work/syntax_suggest/syntax_suggest/vendor/bundle/ruby/3.2.0/bin/rake)
/opt/hostedtoolcache/Ruby/3.2.1/x64/lib/ruby/gems/3.2.0/gems/bundler-2.3.14/lib/bundler/runtime.rb:309:in `check_for_activated_spec!': You have already activated syntax_suggest 1.0.2, but your Gemfile requires syntax_suggest 1.0.3. Since syntax_suggest is a default gem, you can either remove your dependency on it or try updating to a newer version of bundler that supports syntax_suggest as a default gem. (Gem::LoadError)
from /opt/hostedtoolcache/Ruby/3.2.1/x64/lib/ruby/gems/3.2.0/gems/bundler-2.3.14/lib/bundler/runtime.rb:25:in `block in setup'
from /opt/hostedtoolcache/Ruby/3.2.1/x64/lib/ruby/gems/3.2.0/gems/bundler-2.3.14/lib/bundler/spec_set.rb:138:in `each'
from /opt/hostedtoolcache/Ruby/3.2.1/x64/lib/ruby/gems/3.2.0/gems/bundler-2.3.14/lib/bundler/spec_set.rb:138:in `each'
from /opt/hostedtoolcache/Ruby/3.2.1/x64/lib/ruby/gems/3.2.0/gems/bundler-2.3.14/lib/bundler/runtime.rb:24:in `map'
from /opt/hostedtoolcache/Ruby/3.2.1/x64/lib/ruby/gems/3.2.0/gems/bundler-2.3.14/lib/bundler/runtime.rb:24:in `setup'
from /opt/hostedtoolcache/Ruby/3.2.1/x64/lib/ruby/gems/3.2.0/gems/bundler-2.3.14/lib/bundler.rb:151:in `setup'
from /opt/hostedtoolcache/Ruby/3.2.1/x64/lib/ruby/gems/3.2.0/gems/bundler-2.3.14/lib/bundler/setup.rb:20:in `block in <top (required)>'
from /opt/hostedtoolcache/Ruby/3.2.1/x64/lib/ruby/gems/3.2.0/gems/bundler-2.3.14/lib/bundler/ui/shell.rb:136:in `with_level'
from /opt/hostedtoolcache/Ruby/3.2.1/x64/lib/ruby/gems/3.2.0/gems/bundler-2.3.14/lib/bundler/ui/shell.rb:88:in `silence'
from /opt/hostedtoolcache/Ruby/3.2.1/x64/lib/ruby/gems/3.2.0/gems/bundler-2.3.14/lib/bundler/setup.rb:20:in `<top (required)>'
from /opt/hostedtoolcache/Ruby/3.2.1/x64/lib/ruby/gems/3.2.0/gems/bundler-2.3.14/lib/bundler/cli/exec.rb:56:in `require_relative'
from /opt/hostedtoolcache/Ruby/3.2.1/x64/lib/ruby/gems/3.2.0/gems/bundler-2.3.14/lib/bundler/cli/exec.rb:56:in `kernel_load'
from /opt/hostedtoolcache/Ruby/3.2.1/x64/lib/ruby/gems/3.2.0/gems/bundler-2.3.14/lib/bundler/cli/exec.rb:23:in `run'
from /opt/hostedtoolcache/Ruby/3.2.1/x64/lib/ruby/gems/3.2.0/gems/bundler-2.3.14/lib/bundler/cli.rb:483:in `exec'
from /opt/hostedtoolcache/Ruby/3.2.1/x64/lib/ruby/gems/3.2.0/gems/bundler-2.3.14/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
```
syntax_suggest did not work great when there is no new line at the end
of the input file.
Input:
```
def foo
end
end # No newline at end of file
```
Previous output:
```
$ ruby test.rb
test.rb: --> test.rb
Unmatched `end', missing keyword (`do', `def`, `if`, etc.) ?
> 1 def foo
> 2 end
> 3 end # No newline at end of filetest.rb:3: syntax error, unexpected `end' (SyntaxError)
end # No newline at end of file
^~~
```
Note that "test.rb:3: ..." is appended to the last line of the
annotation.
This change makes sure that the annotation ends with a new line.
New output:
```
$ ruby test.rb
test.rb: --> test.rb
Unmatched `end', missing keyword (`do', `def`, `if`, etc.) ?
> 1 def foo
> 2 end
> 3 end # No newline at end of file
test.rb:3: syntax error, unexpected `end' (SyntaxError)
end # No newline at end of file
^~~
```
https://github.com/ruby/syntax_suggest/commit/db4cf9147d
Originally I fixed https://github.com/ruby/syntax_suggest/pull/177 by making the process of comment removal indentation aware. The next commit is the more general fix and means we don't need to carry that additional logic/overhead.
Also: Update syntax via linter
While #177 is reported as being caused by a comment, the underlying behavior is a problem due to the newline that we generated (from a comment). The prior commit fixed that problem by preserving whitespace before the comment. That guarantees that a block will form there from the frontier before it will be expanded there via a "neighbors" method. Since empty lines are valid ruby code, it will be hidden and be safe.
## Problem setup
This failure mode is not fixed by the prior commit, because the indentation is 0. To provide good results, we must make the algorithm less greedy. One heuristic/signal to follow is developer added newlines. If a developer puts a newline between code, it's more likely they're unrelated. For example:
```
port = rand(1000...9999)
stub_request(:any, "localhost:#{port}")
query = Cutlass::FunctionQuery.new(
port: port
).call
expect(WebMock).to have_requested(:post, "localhost:#{port}").
with(body: "{}")
```
This code is split into three chunks by the developer. Each are likely (but not guaranteed) to be intended to stand on their own (in terms of syntax). This behavior is good for scanning neighbors (same indent or higher) within a method, but bad for parsing neighbors across methods.
## Problem
Code is expanded to capture all neighbors, and then it decreases indent level which allows it to capture surrounding scope (think moving from within the method to also capturing the `def/end` definition. Once the indentation level has been increased, we go back to scanning neighbors, but now neighbors also contain keywords.
For example:
```
1 def bark
2
3 end
4
5 def sit
6 end
```
In this case if lines 4, 5, and 6 are in a block when it tries to expand neighbors it will expand up. If it stops after line 2 or 3 it may cause problems since there's a valid kw/end pair, but the block will be checked without it.
TLDR; It's good to stop scanning code after hitting a newline when you're in a method...it causes a problem scanning code between methods when everything inside of one of the methods is an empty line.
In this case it grabs the end on line 3 and since the problem was an extra end, the program now compiles correctly. It incorrectly assumes that the block it captured was causing the problem.
## Extra bit of context
One other technical detail is that after we've decided to stop scanning code for a new neighbor block expansion, we look around the block and grab any empty newlines. Basically adding empty newlines before of after a code block do not affect the parsing of that block.
## The fix
Since we know that this problem only happens when there's a newline inside of a method and we know this particular failure mode is due to having an invalid block (capturing an extra end, but not it's keyword) we have all the metadata we need to detect this scenario and correct it.
We know that the next line above our block must be code or empty (since we grabbed extra newlines). Same for code below it. We can count all the keywords and ends in the block. If they are balanced, it's likely (but not guaranteed) we formed the block correctly. If they're imbalanced, look above or below (depending on the nature of the imbalance), check to see if adding that line would balance the count.
This concept of balance and "leaning" comes from work in https://github.com/ruby/syntax_suggest/pull/152 and has proven useful, but not been formally introduced into the main branch.
## Outcome
Adding this extra check introduced no regressions and fixed the test case. It might be possible there's a mirror or similar problem that we're not handling. That will come out in time. It might also be possible that this causes a worse case in some code not under test. That too would come out in time.
One other possible concern to adding logic in this area (which is a hot codepath), is performance. This extra count check will be performed for every block. In general the two most helpful performance strategies I've found are reducing total number of blocks (therefore reducing overall N internal iterations) and making better matches (the parser to determine if a close block is valid or not is a major bottleneck. If we can split valid code into valid blocks, then it's only evaluated by the parser once, where as invalid code must be continuously re-checked by the parser until it becomes valid, or is determined to be the cause of the core problem.
This extra logic should very rarely result in a change, but when it does it should tend to produce slightly larger blocks (by one line) and more accurate blocks.
Informally it seems to have no impact on performance:
``
This branch:
DEBUG_DISPLAY=1 bundle exec rspec spec/ --format=failures 3.01s user 1.62s system 113% cpu 4.076 total
```
```
On main:
DEBUG_DISPLAY=1 bundle exec rspec spec/ --format=failures 3.02s user 1.64s system 113% cpu 4.098 total
```
https://github.com/ruby/syntax_suggest/commit/13739c6946
When removing comments I previously replaced them with a newline. This loses some context and may affect the order of the indent search which in turn affects the final result. By preserving whitespace in front of the comment, we preserve the "natural" indentation order of the line while also allowing the parser/lexer to see and join naturally consecutive (method chain) lines.
close https://github.com/ruby/syntax_suggest/pull/177
When `tmp/alias` already exists, I'm now getting phantom folders in the directory pointing at older aliases which is distracting/confusing. By checking and removing that alias before symlinking we can prevent this strange behavior (possibly caused by newer Mac OS?).
While working on locking multiple platforms by default, I got an
infinite resolution loop in one of our resolver specs.
The culprit ended up being that when dealing with lockfile specs with
incomplete dependencies (spec appears in lockfile, but its dependencies
don't), those specs were not being properly expired and that tripped up
resolution.
The issue for some reason only manifests when dealing with multiple
lockfile platforms, that's why it only manifested when working on
locking multiple platforms by default.
https://github.com/rubygems/rubygems/commit/4ca72913bb
* Test existing behavior
Typing Ctrl-D ends editing but typing <Del> does not.
Also renamed a test that is not testing ed_delete_next_char but
key_delete.
* Check if line empty first in em_delete
By distributivity of AND over OR, we can factor out this condition. This
will make the next commit simpler.
* Use em_delete in key_delete
When the editing mode is emacs, use `em_delete` in `key_delete`. We need
to add a condition though to `em_delete`, because it implements both
`delete-char` and `end-of-file`. We only want the `end-of-file` behavior
is the key is really Ctrl-D.
This matches the behavior of the <Del> key with readline, i.e. deleting
the next character if there is one, but not moving the cursor, while not
finishing the editing if there are no characters.
Remove !USE_RVARGC code
[Feature #19579]
The Variable Width Allocation feature was turned on by default in Ruby
3.2. Since then, we haven't received bug reports or backports to the
non-Variable Width Allocation code paths, so we assume that nobody is
using it. We also don't plan on maintaining the non-Variable Width
Allocation code, so we are going to remove it.
Ruby implementations like JRuby and TruffleRuby already indicate their
compatibility target with RUBY_VERSION. We don't need to exclude
them from accepting keyword arguments as long as they target 2.7+.
https://github.com/ruby/irb/commit/bf20faa4e6
Co-authored-by: Kevin Menard <kevin@nirvdrum.com>
* Remove unused ATTR_TTY and ATTR_PLAIN constants
They were added in d7d26b51bf
But the references were removed in 1c76845cca
Co-authored-by: Alexandre Terrasa <alexandre.terrasa@shopify.com>
* Remove unused MethodExtender module
It was added in 6cc5d718d7
but it's not used anywhere.
Co-authored-by: Alexandre Terrasa <alexandre.terrasa@shopify.com>
* Remove unused IRB.irb_at_exit
It's not used after aaf4eb4e98
Co-authored-by: Alexandre Terrasa <alexandre.terrasa@shopify.com>
* Remove unused InputCompletor.ignored_modules
It was added in 88311ce3c8
but the reference was removed in 78c74d2425
* Remove unused TracerLoadError constant
This constant was added in cb50fa3738
but never referenced.
---------
https://github.com/ruby/irb/commit/7de0234325
Co-authored-by: Alexandre Terrasa <alexandre.terrasa@shopify.com>
straightforward
(https://github.com/ruby/reline/pull/526)
* Improve test coverage on Unicode.take_range
* Add test for Unicode.calculate_width
* Expand the scanned array to later case statement more straightforward
When dependencies have changed, we'll be re-resolving, and we can't
really know whether the resolution will be valid or invalid for the Ruby
platform, so skip the removal in that case.
The fix worked, but made some other specs fail, and surfaced that the
`@dependencies_changed` attribute was actually being incorrect set when
explicitly unlocking. Fixed that with an early return.
https://github.com/rubygems/rubygems/commit/20d8f5e5d9
Bundler is very conservative by default, trying to preserve versions
from the lockfile as possible, and never downgrading them. However, when
it runs into a resolution error, it still tries to find a valid
resolution.
This fallback behavior was too "brute-force" though, completely
unrestricting any gem found in the resolution conflict, and that could
lead to direct dependencies being downgraded in some edge cases.
Instead, unlock things a bit more carefully:
* First try unlocking fully pinned indirect dependencies, but leave a
lower bound requirement in place to prevent downgrades.
* Then try unlocking any fully pinned dependency, also leaving a lower
bound requirement in place.
* Finally completely unrestrict dependencies if nothing else worked.
https://github.com/rubygems/rubygems/commit/7f55ed8302
Prereleases are NOT considered even when the dependency specification
includes a prerelease segment, if the dependency is not locked. To me
that doesn't make sense, because I could create a Gemfile with say gem
"rails", ">= 7.0.0.beta1" and I'd expect prereleases considered there
even if there's no lock file yet.
Co-authored-by: David Rodríguez <deivid.rodriguez@riseup.net>
I recently ran into very nasty issues with dynamic symbols clashing between
two native gems.
I believe the overwhelming majority of native gems don't want to export
their symbols, so hidding them by default would make sense to me.
https://github.com/rubygems/rubygems/commit/449624aa54
I've never seen this error in real life, and if it was happening, I
think it's either some server side issue that would need to be fixed or
some transient issue. We should move away from the full index, since
it's slow, so let's stop recommending it.
The debug message suggests retrying using `--full-index`, but the retry
is happening automatically. Just log that we are falling back to the
full index, like we do with other errors.
Sometimes we'll have an heterogenous array of specs which include
`Gem::Specification` objects, which don't define `#identifier`. Let's
use `#full_name` consistently.
Recent bugs fixed made me realize we were relying on state too much
here. We only need to keep incomplete specs to be able to expire them
and retry resolution without them locked. If we use a separate class, we
can do that more transparently and handle them just like we handle
"missing specs".
This is just a suggestion, but since recent versions of Ruby have shipped with a `binding.irb` that provides almost all of the features that `pry` pioneered, it may be time to remove this suggestion in the name of simplicity and not confusing newer developers who aren't familiar with what to use.
(https://github.com/ruby/reline/pull/509)
* Add key bindings for PgUp, PgDn
* Match behavior of readline 8.2
In the latest readline (8.2), page-up and page-down are bound to
history-search-backward and history-search-forward by default.
We would like reline to have the same default behavior.
Since Bundler 2.4, we will try to checkout any branch specified in the
Gemfile, while until Bundler 2.3 we would directly checkout the locked
revision.
This should not make any difference in most situations, but in some edge
cases, like if the branch specified in the `Gemfile` has been renamed,
but the locked revision still exist, it causes an error now while before
it would update the lockfile without issues.
I debated which behavior was best, since I was not sure. But my
conclusion is that if the situation does not require expiring the
lockfile source in favor of the Gemfile source, we should use the locked
revision directly and proceed happily. So I restored Bundler 2.3
behavior.
I think this is consistent with how yanked gems are handled, for example.
Of course, if explicitly updating the git source itself, or all gems, we
will still get any errors like missing branches related to the git source.
This was working fine for direct dependencies using
`force_ruby_platform` explicitly through Gemfile, but not for indirect
dependencies. In general, indirect dependencies do not have this
property set, but in truffleruby this is different and the default value
is to have it set.
This should be a very rare edge case, however, it does happen when using
a .dev version of Bundler because in that case, that's the only version
that the resolver considers, and it should not be ignored.
We could've special cased this specifically for Bundler, but I think it
does make sense for every gem.
I've seen this error sometimes but never found it helpful, since the
culprit has never been thread/monitor related, so it just adds noise.
Right now I'm seeing an issue in CI where we seem to be crashing during
ensure and showing a weird backtrace:
```
#<Thread:0x000055821d7a73b8 /home/runner/work/rubygems/rubygems/lib/rubygems/request_set.rb:167 run> terminated with exception (report_on_exception is true):
/home/runner/work/rubygems/rubygems/lib/rubygems/platform.rb:141:in `==': undefined local variable or method `to_a' for false:FalseClass (NameError)
Did you mean? to_s
from /home/runner/work/rubygems/rubygems/lib/rubygems/core_ext/kernel_require.rb:168:in `!='
from /home/runner/work/rubygems/rubygems/lib/rubygems/core_ext/kernel_require.rb:168:in `require'
from /home/runner/work/rubygems/rubygems/lib/rubygems.rb:441:in `block in ensure_subdirectories'
from /home/runner/work/rubygems/rubygems/lib/rubygems.rb:437:in `each'
from /home/runner/work/rubygems/rubygems/lib/rubygems.rb:437:in `ensure_subdirectories'
from /home/runner/work/rubygems/rubygems/lib/rubygems.rb:413:in `ensure_gem_subdirectories'
from /home/runner/work/rubygems/rubygems/lib/rubygems/resolver/specification.rb:110:in `download'
from /home/runner/work/rubygems/rubygems/lib/rubygems/request_set.rb:173:in `block (2 levels) in install'
```
Let's get this out of the picture in case that helps clarify the
culprit.
https://github.com/rubygems/rubygems/commit/87ce40b70e
String#unpack("C*") is equivalent to String#bytes and Array#inject {|a,b| a + b } is equivalent to Array#sum.
This is a minor enhancement in terms of speed and readability.
peek_at_stack was doubly accounting for C::VM_CALL_ARGS_BLOCKARG, which
made that case always side-exit and hide this bug.
When a block handler is block param proxy, you have to set LEP's specval
as next specval instead of block param proxy itself.
When we use this methods with local gemspec, we don't handle
build status of extension correctly. So We need to find extension
files in require_paths.
Example with ruby/erb repository:
```
$ bundle exec irb
Ignoring erb-4.0.2 because its extensions are not built. Try: gem pristine erb --version 4.0.2
>>
```
https://github.com/rubygems/rubygems/commit/f90e43cf3f
Hiroshi SHIBATA
Aaron Patterson
tomoya ishida
卜部昌平
John Hawthorn
Marcin Rozmus
Ashley Ellis Pierce
Eric Herscovich
Jenny Shen
ima1zumi
Kasumi Hanazuki
Stan Lo
David Rodríguez
schneems
Yusuke Endoh
Nobuyoshi Nakada
Vít Ondruch
Takashi Kokubun
Étienne Barrié
Jeremy Evans
Peter Zhu
git
NAKAMURA Usaku
Josef Šimánek
Michael Siegfried
Jean Boussier
Mau Magnaguagno
なつき
Justin Searls
Phillip Hellewell
Ian Ker-Seymer