If we're in inline mode, Bundler first resolves using only local gems,
and if some gems are missing, then it re-resolves using remote gems.
However, "source resolution" from the initial "local" try was being
memoized, resulting in Bundler not looking for some gems remotely in the
second resolution.
This commit forces a proper re-resolve in this case.
https://github.com/rubygems/rubygems/commit/fdc631075e
* Add rb_io_path and rb_io_open_descriptor.
* Use rb_io_open_descriptor to create PTY objects
* Rename FMODE_PREP -> FMODE_EXTERNAL and expose it
FMODE_PREP I believe refers to the concept of a "pre-prepared" file, but
FMODE_EXTERNAL is clearer about what the file descriptor represents and
aligns with language in the IO::Buffer module.
* Ensure that rb_io_open_descriptor closes the FD if it fails
If FMODE_EXTERNAL is not set, then it's guaranteed that Ruby will be
responsible for closing your file, eventually, if you pass it to
rb_io_open_descriptor, even if it raises an exception.
* Rename IS_EXTERNAL_FD -> RUBY_IO_EXTERNAL_P
* Expose `rb_io_closed_p`.
* Add `rb_io_mode` to get IO mode.
---------
Co-authored-by: KJ Tsanaktsidis <ktsanaktsidis@zendesk.com>
[Feature #19236]
In Ruby 3.3, `Hash.new` shall print a deprecation warning if keyword arguments
are passed instead of treating them as an implicit positional Hash.
This will allow to safely introduce a `capacity` keyword argument in 3.4
Co-authored-by: Jean Boussier <byroot@ruby-lang.org>
https://github.com/ruby/syntax_suggest/pull/187 Handle if/else with
empty/comment
line
Reported in #187 this code:
```
class Foo
def foo
if cond?
foo
else
# comment
end
end
# ...
def bar
if @recv
end_is_missing_here
end
end
```
Triggers an incorrect suggestion:
```
Unmatched keyword, missing `end' ?
1 class Foo
2 def foo
> 3 if cond?
> 5 else
8 end
16 end
```
Part of the issue is that while scanning we're using newlines to determine when to stop and pause. This is useful for determining logically smaller chunks to evaluate but in this case it causes us to pause before grabbing the "end" that is right below the newline. This problem is similar to https://github.com/ruby/syntax_suggest/pull/179.
However in the case of expanding same indentation "neighbors" I want to always grab all empty values at the end of the scan. I don't want to do that when changing indentation levels as it affects scan results.
There may be some way to normalize this behavior between the two, but the tests really don't like that change.
To fix this issue for expanding against different indentation I needed a way to first try and grab any additional newlines with the ability to rollback that guess. In #192 I experimented with decoupling scanning from the AroundBlockScan logic. It also added the ability to take a snapshot of the current scanner and rollback to prior changes.
With this ability in place now we:
- Grab extra empties before looking at the above/below line for the matching keyword/end statement
- If there's a match, grab it
- If there's no match, discard the newlines we picked up in the evaluation
That solves the issue.
https://github.com/ruby/syntax_suggest/commit/513646b912
AroundBlockScan started as a utility class that was meant to be used as a DSL for scanning and making new blocks. As logic got added to this class it became hard to reason about what exactly is being mutated when. I pulled the scanning logic out into it's own class which gives us a clean separation of concerns. This allowed me to remove a lot of accessors that aren't core to the logic provided by AroundBlockScan.
In addition to this refactor the ScanHistory class can snapshot a scan. This allows us to be more aggressive with scans in the future as we can now snapshot and rollback if it didn't turn out the way we wanted.
The change comes with a minor perf impact:
before: 5.092678 0.104299 5.196977 ( 5.226494)
after: 5.128536 0.099871 5.228407 ( 5.249542)
This represents a 0.996x change in speed (where 1x would be no change and 2x would be twice as fast). This is a 0.38% decrease in performance which is negligible. It's likely coming from the extra blocks being created while scanning. This is negligible and if the history feature works well we might be able to make better block decisions which is means fewer calls to ripper which is the biggest bottleneck.
While this doesn't fix https://github.com/ruby/syntax_suggest/issues/187 it's a good intermediate step that will hopefully make working on that issue easier.
https://github.com/ruby/syntax_suggest/commit/ad8487d8aa
I previously left a comment stating I didn't know why a certain method existed. In investigating the code in `CaptureCodeContext#capture_before_after_kws` I found that it was added as to give a slightly less noisy output.
The docs for AroundBlockScan#capture_neighbor_context only describe keywords as being a primary concern. I modified that code to only include lines that are keywords or ends. This reduces the output noise even more.
This allows me to remove that `start_at_next_line` method.
One weird side effect of the prior logic is it would cause this code to produce this output:
```
class OH
def hello
def hai
end
end
```
```
1 class OH
> 2 def hello
4 def hai
5 end
6 end
```
But this code to produce this output:
```
class OH
def hello
def hai
end
end
```
```
1 class OH
> 2 def hello
4 end
5 end
```
Note the missing `def hai`. The only difference between them is that space.
With this change, they're now both consistent.
https://github.com/ruby/syntax_suggest/commit/4a54767a3e
`GETRUSAGE_BASED_CLOCK_PROCESS_CPUTIME_ID` clock uses `getrusage`
always if available as the name states. That is if it is implemented
`getrusage` is available, regardless microseconds in its results.
Prior to commit 5806c54447, it was "at
least one result with precision beyond milliseconds (with none-zero
microseconds) should exist"; after this commit, "at least one result
should have zero microseconds". This chance is lower than the
previous condition.
Since these files are written in a wide character encoding, stop at
first NUL byte and are actually empty. ASCII-incompatible encodings
have never been supported as source encoding.
If Gemfile has a lot of dependencies, we have an optimization that uses
the full index in that case, assuming it's going to be faster.
I think this is an old optimization that predates compact index API
times, I believe we no longer need it these days.
Also, since a few releases ago we check for circular dependencies when
resolving by looping through all versions of each name and removing
those that have circular dependencies that would trip up the resolver.
This loop becomes actually very slow when full indexes are used because
to find dependencies of a gemspec, we need to explicitly fetch the
marshaled gemspec (`gemspec.rz` endpoint) for it, so the optimization
has the opposite effect of making things very slow.
https://github.com/rubygems/rubygems/commit/2f46289bd3
Fix a CI error and add a test to ensure we're testing the current version:
```
Run bundle exec rake test
bundler: failed to load command: rake (/home/runner/work/syntax_suggest/syntax_suggest/vendor/bundle/ruby/3.2.0/bin/rake)
/opt/hostedtoolcache/Ruby/3.2.1/x64/lib/ruby/gems/3.2.0/gems/bundler-2.3.14/lib/bundler/runtime.rb:309:in `check_for_activated_spec!': You have already activated syntax_suggest 1.0.2, but your Gemfile requires syntax_suggest 1.0.3. Since syntax_suggest is a default gem, you can either remove your dependency on it or try updating to a newer version of bundler that supports syntax_suggest as a default gem. (Gem::LoadError)
from /opt/hostedtoolcache/Ruby/3.2.1/x64/lib/ruby/gems/3.2.0/gems/bundler-2.3.14/lib/bundler/runtime.rb:25:in `block in setup'
from /opt/hostedtoolcache/Ruby/3.2.1/x64/lib/ruby/gems/3.2.0/gems/bundler-2.3.14/lib/bundler/spec_set.rb:138:in `each'
from /opt/hostedtoolcache/Ruby/3.2.1/x64/lib/ruby/gems/3.2.0/gems/bundler-2.3.14/lib/bundler/spec_set.rb:138:in `each'
from /opt/hostedtoolcache/Ruby/3.2.1/x64/lib/ruby/gems/3.2.0/gems/bundler-2.3.14/lib/bundler/runtime.rb:24:in `map'
from /opt/hostedtoolcache/Ruby/3.2.1/x64/lib/ruby/gems/3.2.0/gems/bundler-2.3.14/lib/bundler/runtime.rb:24:in `setup'
from /opt/hostedtoolcache/Ruby/3.2.1/x64/lib/ruby/gems/3.2.0/gems/bundler-2.3.14/lib/bundler.rb:151:in `setup'
from /opt/hostedtoolcache/Ruby/3.2.1/x64/lib/ruby/gems/3.2.0/gems/bundler-2.3.14/lib/bundler/setup.rb:20:in `block in <top (required)>'
from /opt/hostedtoolcache/Ruby/3.2.1/x64/lib/ruby/gems/3.2.0/gems/bundler-2.3.14/lib/bundler/ui/shell.rb:136:in `with_level'
from /opt/hostedtoolcache/Ruby/3.2.1/x64/lib/ruby/gems/3.2.0/gems/bundler-2.3.14/lib/bundler/ui/shell.rb:88:in `silence'
from /opt/hostedtoolcache/Ruby/3.2.1/x64/lib/ruby/gems/3.2.0/gems/bundler-2.3.14/lib/bundler/setup.rb:20:in `<top (required)>'
from /opt/hostedtoolcache/Ruby/3.2.1/x64/lib/ruby/gems/3.2.0/gems/bundler-2.3.14/lib/bundler/cli/exec.rb:56:in `require_relative'
from /opt/hostedtoolcache/Ruby/3.2.1/x64/lib/ruby/gems/3.2.0/gems/bundler-2.3.14/lib/bundler/cli/exec.rb:56:in `kernel_load'
from /opt/hostedtoolcache/Ruby/3.2.1/x64/lib/ruby/gems/3.2.0/gems/bundler-2.3.14/lib/bundler/cli/exec.rb:23:in `run'
from /opt/hostedtoolcache/Ruby/3.2.1/x64/lib/ruby/gems/3.2.0/gems/bundler-2.3.14/lib/bundler/cli.rb:483:in `exec'
from /opt/hostedtoolcache/Ruby/3.2.1/x64/lib/ruby/gems/3.2.0/gems/bundler-2.3.14/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
```
Originally I fixed https://github.com/ruby/syntax_suggest/pull/177 by making the process of comment removal indentation aware. The next commit is the more general fix and means we don't need to carry that additional logic/overhead.
Also: Update syntax via linter
While #177 is reported as being caused by a comment, the underlying behavior is a problem due to the newline that we generated (from a comment). The prior commit fixed that problem by preserving whitespace before the comment. That guarantees that a block will form there from the frontier before it will be expanded there via a "neighbors" method. Since empty lines are valid ruby code, it will be hidden and be safe.
## Problem setup
This failure mode is not fixed by the prior commit, because the indentation is 0. To provide good results, we must make the algorithm less greedy. One heuristic/signal to follow is developer added newlines. If a developer puts a newline between code, it's more likely they're unrelated. For example:
```
port = rand(1000...9999)
stub_request(:any, "localhost:#{port}")
query = Cutlass::FunctionQuery.new(
port: port
).call
expect(WebMock).to have_requested(:post, "localhost:#{port}").
with(body: "{}")
```
This code is split into three chunks by the developer. Each are likely (but not guaranteed) to be intended to stand on their own (in terms of syntax). This behavior is good for scanning neighbors (same indent or higher) within a method, but bad for parsing neighbors across methods.
## Problem
Code is expanded to capture all neighbors, and then it decreases indent level which allows it to capture surrounding scope (think moving from within the method to also capturing the `def/end` definition. Once the indentation level has been increased, we go back to scanning neighbors, but now neighbors also contain keywords.
For example:
```
1 def bark
2
3 end
4
5 def sit
6 end
```
In this case if lines 4, 5, and 6 are in a block when it tries to expand neighbors it will expand up. If it stops after line 2 or 3 it may cause problems since there's a valid kw/end pair, but the block will be checked without it.
TLDR; It's good to stop scanning code after hitting a newline when you're in a method...it causes a problem scanning code between methods when everything inside of one of the methods is an empty line.
In this case it grabs the end on line 3 and since the problem was an extra end, the program now compiles correctly. It incorrectly assumes that the block it captured was causing the problem.
## Extra bit of context
One other technical detail is that after we've decided to stop scanning code for a new neighbor block expansion, we look around the block and grab any empty newlines. Basically adding empty newlines before of after a code block do not affect the parsing of that block.
## The fix
Since we know that this problem only happens when there's a newline inside of a method and we know this particular failure mode is due to having an invalid block (capturing an extra end, but not it's keyword) we have all the metadata we need to detect this scenario and correct it.
We know that the next line above our block must be code or empty (since we grabbed extra newlines). Same for code below it. We can count all the keywords and ends in the block. If they are balanced, it's likely (but not guaranteed) we formed the block correctly. If they're imbalanced, look above or below (depending on the nature of the imbalance), check to see if adding that line would balance the count.
This concept of balance and "leaning" comes from work in https://github.com/ruby/syntax_suggest/pull/152 and has proven useful, but not been formally introduced into the main branch.
## Outcome
Adding this extra check introduced no regressions and fixed the test case. It might be possible there's a mirror or similar problem that we're not handling. That will come out in time. It might also be possible that this causes a worse case in some code not under test. That too would come out in time.
One other possible concern to adding logic in this area (which is a hot codepath), is performance. This extra count check will be performed for every block. In general the two most helpful performance strategies I've found are reducing total number of blocks (therefore reducing overall N internal iterations) and making better matches (the parser to determine if a close block is valid or not is a major bottleneck. If we can split valid code into valid blocks, then it's only evaluated by the parser once, where as invalid code must be continuously re-checked by the parser until it becomes valid, or is determined to be the cause of the core problem.
This extra logic should very rarely result in a change, but when it does it should tend to produce slightly larger blocks (by one line) and more accurate blocks.
Informally it seems to have no impact on performance:
``
This branch:
DEBUG_DISPLAY=1 bundle exec rspec spec/ --format=failures 3.01s user 1.62s system 113% cpu 4.076 total
```
```
On main:
DEBUG_DISPLAY=1 bundle exec rspec spec/ --format=failures 3.02s user 1.64s system 113% cpu 4.098 total
```
https://github.com/ruby/syntax_suggest/commit/13739c6946
When removing comments I previously replaced them with a newline. This loses some context and may affect the order of the indent search which in turn affects the final result. By preserving whitespace in front of the comment, we preserve the "natural" indentation order of the line while also allowing the parser/lexer to see and join naturally consecutive (method chain) lines.
close https://github.com/ruby/syntax_suggest/pull/177
While working on locking multiple platforms by default, I got an
infinite resolution loop in one of our resolver specs.
The culprit ended up being that when dealing with lockfile specs with
incomplete dependencies (spec appears in lockfile, but its dependencies
don't), those specs were not being properly expired and that tripped up
resolution.
The issue for some reason only manifests when dealing with multiple
lockfile platforms, that's why it only manifested when working on
locking multiple platforms by default.
https://github.com/rubygems/rubygems/commit/4ca72913bb
[Feature #18885]
For now, the optimizations performed are:
- Run a major GC
- Compact the heap
- Promote all surviving objects to oldgen
Other optimizations may follow.
When dependencies have changed, we'll be re-resolving, and we can't
really know whether the resolution will be valid or invalid for the Ruby
platform, so skip the removal in that case.
The fix worked, but made some other specs fail, and surfaced that the
`@dependencies_changed` attribute was actually being incorrect set when
explicitly unlocking. Fixed that with an early return.
https://github.com/rubygems/rubygems/commit/20d8f5e5d9
Bundler is very conservative by default, trying to preserve versions
from the lockfile as possible, and never downgrading them. However, when
it runs into a resolution error, it still tries to find a valid
resolution.
This fallback behavior was too "brute-force" though, completely
unrestricting any gem found in the resolution conflict, and that could
lead to direct dependencies being downgraded in some edge cases.
Instead, unlock things a bit more carefully:
* First try unlocking fully pinned indirect dependencies, but leave a
lower bound requirement in place to prevent downgrades.
* Then try unlocking any fully pinned dependency, also leaving a lower
bound requirement in place.
* Finally completely unrestrict dependencies if nothing else worked.
https://github.com/rubygems/rubygems/commit/7f55ed8302
Add tests for pre, move more of the setup into a helper method, and
restructure tests.
There seem to be five considerations for these tests (level, pre, strict,
locked, and whether the current version is a prerelease version, though
the last one overlaps with pre and didn't seem to behave how I expected
under test). Rather than write out the 16 (/32 if the last consideration
is real) combinations, I wrote most with independent tests for each
value. The existing combined tests were maintained (level vs strict)
because these seem the most interrelated.
https://github.com/rubygems/rubygems/commit/74c23a91b2
This is useful for passing directory file descriptors over UNIX
sockets or to child processes to avoid TOCTOU vulnerabilities.
The implementation follows the Dir.chdir code.
This will raise NotImplementedError on platforms not supporting
both fchdir and dirfd.
Implements [Feature #19347]
The documentation states it returns a copy of self with nil value
entries removed. However, the previous behavior was creating a
plain new hash with non-nil values copied into it. This change
aligns the behavior with the documentation.
Fixes [Bug #19113]
This was already copied for non-empty hashes. As Hash.ruby2_keywords_hash
copies default values, it should also copy the compare_by_identity flag.
Partially Fixes [Bug #19113]
It wasn't copied for empty hashes, and Hash.[] doesn't copy the
default value, so copying the compare_by_identity flag does not
make sense.
Partially Fixes [Bug #19113]
I've never seen this error in real life, and if it was happening, I
think it's either some server side issue that would need to be fixed or
some transient issue. We should move away from the full index, since
it's slow, so let's stop recommending it.
The debug message suggests retrying using `--full-index`, but the retry
is happening automatically. Just log that we are falling back to the
full index, like we do with other errors.
Since Bundler 2.4, we will try to checkout any branch specified in the
Gemfile, while until Bundler 2.3 we would directly checkout the locked
revision.
This should not make any difference in most situations, but in some edge
cases, like if the branch specified in the `Gemfile` has been renamed,
but the locked revision still exist, it causes an error now while before
it would update the lockfile without issues.
I debated which behavior was best, since I was not sure. But my
conclusion is that if the situation does not require expiring the
lockfile source in favor of the Gemfile source, we should use the locked
revision directly and proceed happily. So I restored Bundler 2.3
behavior.
I think this is consistent with how yanked gems are handled, for example.
Of course, if explicitly updating the git source itself, or all gems, we
will still get any errors like missing branches related to the git source.
This was working fine for direct dependencies using
`force_ruby_platform` explicitly through Gemfile, but not for indirect
dependencies. In general, indirect dependencies do not have this
property set, but in truffleruby this is different and the default value
is to have it set.
This should be a very rare edge case, however, it does happen when using
a .dev version of Bundler because in that case, that's the only version
that the resolver considers, and it should not be ignored.
We could've special cased this specifically for Bundler, but I think it
does make sense for every gem.
Currently, the --no-install option to `bundle package` is totally
ignored for git sources. This can have very strange effects if you have:
- a git-sourced gem,
- with native extensions,
- whose extconf.rb script depends on another gem,
- which is installed from Rubygems in the gemfile.
In that circumstance, `bundle package --no-install --all` will download
the Rubygems dependencies to `vendor/cache` but NOT install them. It
will also check out the git gems to `vendor/cache` (good), and attempt
to build their native extensions (bad!).
The native extension build will fail because the extconf.rb script crashes,
since the dependency it needs is missing.
I implemented a fix for this in `source/git.rb`, since this is analogous
to what's happening in `source/rubygems.rb`. I do admit though the whole
thing is a little strange though - an "install" method that.... proceeds
to look at a global flag to not install anything.
Add test to confirm cache respects the --no-install flag
https://github.com/rubygems/rubygems/commit/5a77d1c397
Co-authored-by: KJ Tsanaktsidis <kj@kjtsanaktsidis.id.au>
If the original `BUNDLE_GEMFILE` is different from the default, then the
suggestion wouldn't work as is.
Before:
```
$ util/rubocop
Could not find rubocop-1.30.1 in locally installed gems
Run `bundle install` to install missing gems.
$ rubygems git:(better-cmd-suggestion) ✗ bundle install
Could not locate Gemfile
```
After:
```
$ util/rubocop
Could not find rubocop-1.30.1 in locally installed gems
Run `bundle install --gemfile /path/to/rubygems/bundler/tool/bundler/lint_gems.rb` to install missing gems.
$ bundle install --gemfile /path/to/rubygems/bundler/tool/bundler/lint_gems.rb
Fetching gem metadata from https://rubygems.org/.........
Using ast 2.4.2
Using bundler 2.4.7
Using parser 3.1.2.0
Using rainbow 3.1.1
Using parallel 1.22.1
Using regexp_parser 2.5.0
Using rubocop-ast 1.18.0
Using rexml 3.2.5
Using ruby-progressbar 1.11.0
Using unicode-display_width 2.1.0
Fetching rubocop 1.30.1
Installing rubocop 1.30.1
Using rubocop-performance 1.14.2
Bundle complete! 2 Gemfile dependencies, 12 gems now installed.
Use `bundle info [gemname]` to see where a bundled gem is installed.
$ util/rubocop
Inspecting 345 files
.........................................................................................................................................................................................................................................................................................................................................................
345 files inspected, no offenses detected
```
https://github.com/rubygems/rubygems/commit/bf1320d805
Following up on https://github.com/rubygems/rubygems/pull/6355, which
turned a crash into a nicer error message, this commit auto-heals the
corrupt lockfile instead.
In this particular case (a corrupt Gemfile.lock with missing
dependencies) the LazySpecification will not have accurate dependency
information, we have to materialize the SpecSet to determine there are
missing dependencies. We've already got a way to handle this, via
`SpecSet#incomplete_specs`, but it wasn't quite working for this case
because we'd get to `@incomplete_specs += lookup[name]` and
`lookup[name]` would be empty for the dependency.
With this commit we catch it a bit earlier, marking the parent spec
containing the missing dependency as incomplete.
https://github.com/rubygems/rubygems/commit/486ecb8f20
[Bug #19439]
The instance variables were restore on the Regexp source,
not the regexp itself.
Unfortunately we have a bit of a chicken and egg problem.
The source holds the encoding, and the encoding need to be set on
the source to be able to instantiate the Regexp.
So the instance variables have to be read on the `source`.
To correct this we transfert the instance variables after
instantiating the Regexp.
The only way to avoid this would be to read the instance variable
twice and rewind.
* Replaces the wording of "is forbidden" with "cannot be used"
* Fixes the method signature of VersionRange::Empty#eql?
https://github.com/rubygems/rubygems/commit/8c6b3f130b
Co-authored-by: Daniel Colson <danieljamescolson@gmail.com>
This test is no longer passing:
```
1)
BigDecimal#remainder returns NaN if Infinity is involved FAILED
Expected Infinity.nan?
to be truthy but was false
/home/runner/work/ruby/ruby/src/spec/ruby/library/bigdecimal/remainder_spec.rb:58:in `block (2 levels) in <top (required)>'
/home/runner/work/ruby/ruby/src/spec/ruby/library/bigdecimal/remainder_spec.rb:4:in `<top (required)>'
```
https://github.com/ruby/bigdecimal/pull/243
I did a bad thing (script that edits the Gemfile.lock directly) and
ended up with a Gemfile.lock that was completely missing some indirect
dependencies. While this is my fault and an error is reasonable, I
noticed that the error got progressively less friendly in recent
versions of bundler.
Something similar came up in https://github.com/rubygems/rubygems/issues/6210,
and this commit would have helped with that case as well
(although we've already handled this a different way with #6219).
Details:
---
Back on Bundler 2.2.23, a corrupt lockfile like this would cause a helpful error:
```
Unable to find a spec satisfying minitest (>= 5.1) in the set. Perhaps the lockfile is corrupted?
```
Bundler 2.3.26 gave a helpful warning:
```
Warning:
Your lockfile was created by an old Bundler that left some things out.
Because of the missing DEPENDENCIES, we can only install gems one at a time,
instead of installing 16 at a time.
You can fix this by adding the missing gems to your Gemfile, running bundle
install, and then removing the gems from your Gemfile.
The missing gems are:
* minitest depended upon by activesupport
```
But then continued on and crashed while trying to report the unmet
dependency:
```
--- ERROR REPORT TEMPLATE -------------------------------------------------------
NoMethodError: undefined method `full_name' for nil:NilClass
lib/bundler/installer/parallel_installer.rb:127:in `block (2 levels) in check_for_unmet_dependencies'
...
```
Bundler 2.4.0 and up crash as above when jobs=1, but crash
even harder when run in parallel:
```
--- ERROR REPORT TEMPLATE -------------------------------------------------------
fatal: No live threads left. Deadlock?
3 threads, 3 sleeps current:0x00007fa6b6704660 main thread:0x00007fa6b6704660
* #<Thread:0x000000010833b130 sleep_forever>
rb_thread_t:0x00007fa6b6704660 native:0x0000000108985600 int:0
* #<Thread:0x0000000108dea630@Parallel Installer Worker #0 tmp/1/gems/system/gems/bundler-2.5.0.dev/lib/bundler/worker.rb:90 sleep_forever>
rb_thread_t:0x00007fa6b67f67c0 native:0x0000700009a62000 int:0
* #<Thread:0x0000000108dea4a0@Parallel Installer Worker #1 tmp/1/gems/system/gems/bundler-2.5.0.dev/lib/bundler/worker.rb:90 sleep_forever>
rb_thread_t:0x00007fa6b67f63c0 native:0x0000700009c65000 int:0
<internal:thread_sync>:18:in `pop'
tmp/1/gems/system/gems/bundler-2.5.0.dev/lib/bundler/worker.rb:42:in `deq'
...
```
Changes
---
This commit fixes the confusing thread deadlock crash by detecting if
dependencies are missing such that we'll never be able to enqueue. When
that happens we treat it as a failure so the install can finish.
That gets us back to the `NoMethodError`, which this commit fixes by
using a different warning in the case where no spec is found.
https://github.com/rubygems/rubygems/commit/d73001a21d
`trying to manually editing` doesn't seem quite grammatically
correct. We could change it to `trying to manually edit` (is that a
split infinitive?), but I don't think `trying to` adds much here so
I've removed it instead so `editing` is the verb.
For the list of dependencies, the wording before this commit seemed to
reverse the dependency. "B, depended on A" sounds like B depends on A
(or did in the past but doesn't anymore?), but that's not correct. I
think there's a missing word: "B, depended on by A", but I find "B,
dependency of A" a bit nicer.
https://github.com/rubygems/rubygems/commit/49a31257e3
[Bug #19415]
If multiple threads attemps to load the same file concurrently
it's not a circular dependency issue.
So we check that the existing ThreadShield is owner by the current
fiber before warning about circular dependencies.
[Bug #19415]
If multiple threads attemps to load the same file concurrently
it's not a circular dependency issue.
So we check that the existing ThreadShield is owner by the current
fiber before warning about circular dependencies.
The previous code loads bundler's gemspec which does not include the generated
gemspec file, and thus the test was passing where it should indeed fail.
With this change, the test properly fails now.
https://github.com/rubygems/rubygems/commit/2098ea0d75
Given an existing application using native gems (e.g., nokogiri)
And a lockfile generated with a stable ruby version
When we test the application against ruby-head and `bundle install`
Then bundler should fall back to the generic ruby platform gem
Note that this test has been passing since 45931ac9
https://github.com/rubygems/rubygems/commit/0ecc6de378
Prior to this commit `bundle binstubs --standalone --all` would output a
warning about not being able to generate a standalone binstub for
bundler.
This warning predates the `--all` option, and I don't think it makes
sense in this context. The warning makes good sense when explicitly
trying to generate a bundler standalone binstub with `bundle binstubs
bundler --standalone`, since that command won't do what the user might
have expected. But `--all` is not specifically asking for bundler, and
having it report each time that the bundler binstubs could not be
generated does not seem particularly helpful. The only way to make that
warning go away would be to stop using `--standalone --all`.
This commit skips the warning when running with the `--all` option.
https://github.com/rubygems/rubygems/commit/e6a72e19eb
This commit moves the classpath (and tmp_classpath) from instance
variables to the rb_classext_t. This improves performance as we no
longer need to set an instance variable when assigning a classpath to
a class.
I benchmarked with the following script:
```ruby
name = :MyClass
puts(Benchmark.measure do
10_000_000.times do |i|
Object.const_set(name, Class.new)
Object.send(:remove_const, name)
end
end)
```
Before this patch:
```
5.440119 0.025264 5.465383 ( 5.467105)
```
After this patch:
```
4.889646 0.028325 4.917971 ( 4.942678)
```
Running the file with shebang has a few issues.
* shebang is an OS dependent feature. Many modern UNIX-like OSes
support it, but not all, e.g., Windows.
* `env` command may not be in `/usr/bin`.
* "ruby" command may not be "ruby", when `--program-suffix` or other
configuration option is used.
https://github.com/ruby/syntax_suggest/commit/2edf241055
This was already the behavior when a single `'external:internal'`
encoding specifier string was passed. This makes the behavior
consistent for the case where separate external and internal
encoding specifiers are provided.
While here, fix the IO#set_encoding method documentation to
state that either the first or second argument can be a string
with an encoding name, and describe the behavior when the
external encoding is binary.
Fixes [Bug #18899]
HOMEPATH is set to "\WINDOWS\system32" when running per "runas" session.
This directory is not writable by ordinary users, leading to errors with many ruby tools.
Also config files in the home directory are not recognized.
Still keeping HOME at first which is not used by native Windows,
but by ruby specs and by MSYS2 environment.
Dir.home returns an UTF-8 string since ruby-3.0, but the actual
encoding of the bytes was CP_ACP or CP_OEMCP.
That led to invalid bytes when calling Dir.home with an unicode
username.
Previously, only certain values of the 3rd argument triggered a
deprecation warning.
First step for fix for bug #18797. Support for the 3rd argument
will be removed after the release of Ruby 3.2.
Fix minor fallout discovered by the tests.
Co-authored-by: Nobuyoshi Nakada <nobu@ruby-lang.org>
With `GemVersionPromoter#sort_versions` being so simple, we no longer
need to reach into the class's internals to make private methods public
in order to effectively test. We can just allow both cases to go through
the main method.
https://github.com/rubygems/rubygems/commit/6cbe891003
We have a check for a corrupt lockfile right before installing. However,
the check accounted for locked specs not satisfying locked dependencies,
but not for locked specs missing for some locked dependencies.
Instead of fixing this check, I decided to remove it in favor of
automatically detecting the situation and re-resolve to automatically
fix the lockfile rather than printing a warning but leave the problem
there.
https://github.com/rubygems/rubygems/commit/4a7a584252
bundle lock --update can do everything that bundle update can do, but
it doesn't actually install gems. This is especially useful for
generating a lockfile on a machine that doesn't have the libraries
available to be able to build native extensions.
But, there was no parallel for bundle update --bundler. So let's add
one.
https://github.com/rubygems/rubygems/commit/7fc00bd2a5
Cases like this:
```ruby
obj = Object.new
loop do
obj.instance_variable_set(:@foo, 1)
obj.remove_instance_variable(:@foo)
end
```
can cause us to use many more shapes than we want (and even run out).
This commit changes the code such that when an instance variable is
removed, we'll walk up the shape tree, find the shape, then rebuild any
child nodes that happened to be below the "targetted for removal" IV.
This also requires moving any instance variables so that indexes derived
from the shape tree will work correctly.
Co-Authored-By: Jemma Issroff <jemmaissroff@gmail.com>
Co-authored-by: John Hawthorn <jhawthorn@github.com>
It can be configured by `--with-os-version-style=TYPE` option, and
just copies from theApple's installation as the default. We don't
know why it is major only.
UnboundMethod records caller's class, like `D` or `E` on the
following case:
```ruby
class C
def foo = :foo
end
class D < C
end
class E < C
end
d = D.instance_method(:foo)
e = E.instance_method(:foo)
```
But `d` and `e` only refers `C#foo` so that UnboundMethod doesn't
record `D` or `E`. This behavior changes the following methods:
* `UnboundMethod#inspect` (doesn't show caller's class)
* `UnboundMethod#==` (`d == e` for example)
fix https://bugs.ruby-lang.org/issues/18798
Due to a problem with ripper we do not recognize `break` as invalid code. It's confusing that "Syntax OK" is output in that case.
When there's no syntax error, the algorithm should not say anything. The exception is in the CLI and that's for compatibility with `ruby -wc`
```
$ cat /tmp/break.rb
break
⛄️ 3.1.2 🚀 /Users/rschneeman/Documents/projects/syntax_suggest (schneems/no-syntax-not-okay-break)
$ ruby -wc /tmp/break.rb
Syntax OK
```
> Note that this is invalid, running this code will raise a Syntax error.
```
$ exe/syntax_suggest /tmp/break.rb
Syntax OK
```
Close https://github.com/ruby/syntax_suggest/pull/157https://github.com/ruby/syntax_suggest/commit/d7bd8f03a2
It's questionable whether we want to allow rstrip to work for strings
where the broken coderange occurs before the trailing whitespace and
not after, but this approach is probably simpler, and I don't think
users should expect string operations like rstrip to work on broken
strings.
In some cases, this changes rstrip to raise
Encoding::CompatibilityError instead of ArgumentError. However, as
the problem is related to an encoding issue in the receiver, and due
not due to an issue with an argument, I think
Encoding::CompatibilityError is the more appropriate error.
Fixes [Bug #18931]
Ruby interpreter loads some special gems at startup: did_you_mean,
error_highlight, and syntax_suggest. These gems are loaded before
`bundler/setup` is loaded by `RUBYOPT=-rbundler/setup`.
So, the versions of the gems are not controllable by Gemfile.
This change will `require "bundler/setup"` in rubygems.rb (i.e., before
the special gems are loaded). Now `bundle exec` sets an environment
variable `BUNDLER_SETUP`, and rubygems requires the variable if defined.
See also: https://bugs.ruby-lang.org/issues/19089https://github.com/rubygems/rubygems/commit/963cb65a2d
Not only powerpc64le, also s390x and arm32 seem failing too. These
failures are probably caused by filesystem settings on Travis, but
unrelated to CPUs.
`Complex.polar` accepts Complex values as arguments for the polar form as long
as the value of the complex has no imaginary part (ie it is 'real'). In
`f_complex_polar` this is handled by extracting the real part of the arguments.
However in the case `polar` is called with only a single argument, the absolute
value (abs), then the Complex is created without applying a check on the type
of abs, meaning it is possible to create a Complex where the real part is itself
an instance of a Complex. This change removes the short circuit for the single
argument case meaning the real part extraction is performed correctly
(by f_complex_polar).
Also adds an example to `spec/ruby/core/complex/polar_spec.rb` to check that
the real part of a complex argument is correctly extracted and used in the
resulting Complex real and imaginary parts.
Implements [Feature #12084]
Returns the object for which the receiver is the singleton class, or
raises TypeError if the receiver is not a singleton class.
[Feature #18982]
Instead of introducing an `exception: false` argument to have `non_block`
return nil rather than raise, we can clearly document that a timeout of 0
immediately returns.
The code is refactored a bit to avoid doing a time calculation in
such case.
We sometimes check assertions on lockfile contents, which involves
comparing a reasonably long string. Sometimes RSpec is not able to show
the part of the string that's actually different, making it hard to
figure out the issue.
Configuring this setting should fix the issue in most cases.
https://github.com/rubygems/rubygems/commit/5ad8ee499e
This allows the file to be created without copying permissions
from Bundler's installation source. The previous behaviour was
noticed after installing Ruby through brew, and using bundle
init, which yielded a read-only Gemfile.
https://github.com/rubygems/rubygems/commit/839a06851d
Object Shapes is used for accessing instance variables and representing the
"frozenness" of objects. Object instances have a "shape" and the shape
represents some attributes of the object (currently which instance variables are
set and the "frozenness"). Shapes form a tree data structure, and when a new
instance variable is set on an object, that object "transitions" to a new shape
in the shape tree. Each shape has an ID that is used for caching. The shape
structure is independent of class, so objects of different types can have the
same shape.
For example:
```ruby
class Foo
def initialize
# Starts with shape id 0
@a = 1 # transitions to shape id 1
@b = 1 # transitions to shape id 2
end
end
class Bar
def initialize
# Starts with shape id 0
@a = 1 # transitions to shape id 1
@b = 1 # transitions to shape id 2
end
end
foo = Foo.new # `foo` has shape id 2
bar = Bar.new # `bar` has shape id 2
```
Both `foo` and `bar` instances have the same shape because they both set
instance variables of the same name in the same order.
This technique can help to improve inline cache hits as well as generate more
efficient machine code in JIT compilers.
This commit also adds some methods for debugging shapes on objects. See
`RubyVM::Shape` for more details.
For more context on Object Shapes, see [Feature: #18776]
Co-Authored-By: Aaron Patterson <tenderlove@ruby-lang.org>
Co-Authored-By: Eileen M. Uchitelle <eileencodes@gmail.com>
Co-Authored-By: John Hawthorn <john@hawthorn.email>
Object Shapes is used for accessing instance variables and representing the
"frozenness" of objects. Object instances have a "shape" and the shape
represents some attributes of the object (currently which instance variables are
set and the "frozenness"). Shapes form a tree data structure, and when a new
instance variable is set on an object, that object "transitions" to a new shape
in the shape tree. Each shape has an ID that is used for caching. The shape
structure is independent of class, so objects of different types can have the
same shape.
For example:
```ruby
class Foo
def initialize
# Starts with shape id 0
@a = 1 # transitions to shape id 1
@b = 1 # transitions to shape id 2
end
end
class Bar
def initialize
# Starts with shape id 0
@a = 1 # transitions to shape id 1
@b = 1 # transitions to shape id 2
end
end
foo = Foo.new # `foo` has shape id 2
bar = Bar.new # `bar` has shape id 2
```
Both `foo` and `bar` instances have the same shape because they both set
instance variables of the same name in the same order.
This technique can help to improve inline cache hits as well as generate more
efficient machine code in JIT compilers.
This commit also adds some methods for debugging shapes on objects. See
`RubyVM::Shape` for more details.
For more context on Object Shapes, see [Feature: #18776]
Co-Authored-By: Aaron Patterson <tenderlove@ruby-lang.org>
Co-Authored-By: Eileen M. Uchitelle <eileencodes@gmail.com>
Co-Authored-By: John Hawthorn <john@hawthorn.email>
After recent musl support was added, Bundler started hanging in musl
platforms. I identified the issue where valid candidates were being
filtered out because their platform was specified as a string, and thus
`Gem::Platform.match_spec?` which under the hood ends up calling
`Gem::Platform#===` would return `nil`, because it does not support
comparing platforms to strings.
In particular, `Bundler::EndpointSpecification`'s platform coming from
the API was not instantiated as a `Gem::Platform`, hence the issue.
Also, this spec surfaced another issue where a bug corrected in
`Gem::Platform#match_platforms` had not been yet backported to Bundler.
So this commit also backports that to get the spec green across RubyGems
versions.
Finally, the fix in `Bundler::EndpointSpecification` made a realworld
spec start failing. This spec was faking out `rails-4.2.7.1` requirement
on Bundler in the `Gemfile.lock` file to be `>= 1.17, < 3` when the real
requirement is `>= 1.17, < 2`. Due to the bug in
`Bundler::EndpointSpecification`, the real requirement provided by the
compact index API (recorded with VCR) was being ignored, and the
`Gemfile.lock` fake requirement was being used, which made the spec
pass. This is all expected, and to fix the issue I changed the spec to
be really realworld and don't fake any Bundler requirements.
https://github.com/rubygems/rubygems/commit/faf4ef46bc
Recently a changed was introduced to update the resolver platforms after
it has been created, in order to remove the "ruby" platform from it if
it's to be removed from the lockfile. However, it did not update the
`@resolving_only_for_ruby` instance variable in that case, so the
resolver was not properly doing the right thing anymore.
To fix this, I tweaked the code to restore not changing resolver
platforms after the resolver has been instantiated.
https://github.com/rubygems/rubygems/commit/8fbc30a1d0
When `--conservative` is passed, explicit unlocks are set for top level
gems via `@unlock[:gems]`, so that only those particular gems are
allowed to be updated.
When we compute the "base resolve" from the lockfile (the set of gems
whose versions should be kept pinned by the resolver), we always exclude
gems explicitly unlocked through `@unlock[:gems]` from it. This is done
by the `converge_specs` method.
However, the `converge_specs` method is also used for figuring out
additional lower bound requirements from the lockfile. But in this case,
even if gems are explicitly unlock in `@unlock[:gems]`, we still want to
add the additional requirement, so that gems are not downgraded by the
resolver.
So the solution is to move the line filtering out gems in
`@unlock[:gems]` from the `converged_specs` method out of that method,
so that it only applies for computing the "base resolve", but not the
addtional lower bound requirements.
https://github.com/rubygems/rubygems/commit/405119bd7b
* Fix Array#[] with ArithmeticSequence with negative steps
Previously, Array#[] when called with an ArithmeticSequence
with a negative step did not handle all cases correctly,
especially cases involving infinite ranges, inverted ranges,
and/or exclusive ends.
Fixes [Bug #18247]
* Add Array#slice tests for ArithmeticSequence with negative step to test_array
Add tests of rb_arithmetic_sequence_beg_len_step C-API function.
* Fix ext/-test-/arith_seq/beg_len_step/depend
* Rename local variables
* Fix a variable name
Co-authored-by: Kenta Murata <3959+mrkn@users.noreply.github.com>
Since the change at f310ac1cb2 to show
the backtraces by default, this test started to show the backtraces.
As the backtraces are not the subject of this test, silence them by
using Gem::SilentUI.
This is a regression from a change intended to raise errors when user
puts a gem under an incorrect source in the Gemfile by mistake. To fix
the issue, we revert the change that caused it and implement it in a
different way that restores the resolver independency from real
specifications. Now it deals only with names and versions and does not
try to materialize anything into real specifications before resolving.
https://github.com/rubygems/rubygems/commit/d2bf1b86eb
David Rodríguez
Benoit Daloze
Samuel Williams
NARUSE, Yui
Hiroshi SHIBATA
Peter Zhu
Nobuyoshi Nakada
Jean byroot Boussier
schneems
Ivanov-Anton
Martin Dürst
Jean Boussier
Jeremy Evans
Vladimir Dementyev
git
Jean Boussier
Aaron Patterson
Josef Šimánek
Michael Siegfried
Ian Ker-Seymer
Julie Haehn
Takashi Kokubun
johnnyshields
Tony Hsu
Ellen Marie Dash
Daniel Colson
Mercedes Bernard
John Hawthorn
Yusuke Endoh
Akira Matsuda
Mike Dalessio
Tim Bates
yoka
Kenta Murata
Lars Kanis
Mark Burns
Michael Siegfried
Alyssa Ross
Aaron Patterson
Koichi Sasada
MSP-Greg
Bo Anderson
Stephen Ierodiaconou
Ufuk Kayserilioglu
Jemma Issroff
Victor Gama
Victor Shepelev
Vít Ondruch
Chris Salzberg
Yuta Saito
shields
Aleksandr Varnin
Takuya Noguchi