[Bug #19459] Remove unnecessary always-true checks (#7362)
`length` is a required argument for `IO::Buffer#read` and
`IO::Buffer#write` methods, and `argc` is already checked with
`rb_check_arity`. Also fix the call-seq of `IO::Buffer#read`.
---
io_buffer.c | 24 +++++++-----------------
1 file changed, 7 insertions(+), 17 deletions(-)
Marshal.load: restore instance variables on Regexp
[Bug #19439]
The instance variables were restore on the Regexp source,
not the regexp itself.
Unfortunately we have a bit of a chicken and egg problem.
The source holds the encoding, and the encoding need to be set on
the source to be able to instantiate the Regexp.
So the instance variables have to be read on the `source`.
To correct this we transfert the instance variables after
instantiating the Regexp.
The only way to avoid this would be to read the instance variable
twice and rewind.
---
marshal.c | 20 ++++++++++++++++++--
spec/ruby/core/marshal/shared/load.rb | 11 +++++++++++
2 files changed, 29 insertions(+), 2 deletions(-)
YJIT: jit_prepare_routine_call() for String#+@ missing
We saw SEGVs due to this when running with StackProf, which needs a
correct PC for RUBY_INTERNAL_EVENT_NEWOBJ, the same event used for
ObjectSpace allocation tracing.
[Bug #19444]
---
test/ruby/test_yjit.rb | 27 +++++++++++++++++++++++++++
yjit/src/codegen.rs | 5 ++++-
2 files changed, 31 insertions(+), 1 deletion(-)
YJIT: Fix false assumption that String#+@ => ::String
Could return a subclass.
[Bug #19444]
---
test/ruby/test_yjit.rb | 17 +++++++++++++++++
yjit/src/codegen.rs | 10 +++++++---
2 files changed, 24 insertions(+), 3 deletions(-)
Remove ibf_dumper's WB_PROTECTED status
It doesn't have the right write barriers in place. For example, there is
rb_mark_set(dump->global_buffer.obj_table);
in the mark function, but there is no corresponding write barrier when
adding to the table in the
`ibf_dump_object() -> ibf_table_find_or_insert() -> st_insert()` code path.
To insert write barrier correctly, we need to store the T_STRUCT VALUE
inside `struct ibf_dump`. Instead of doing that, let's just demote it
to WB unproected for correctness. These dumper object are ephemeral so
there is not a huge benefit for having them WB protected.
Users of the bootsnap gem ran into crashes due to this issue:
https://github.com/Shopify/bootsnap/issues/436
Fixes [Bug #19419]
---
compile.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
YJIT: Use the system page size when the code page size is too small
(#7267)
Previously on ARM64 Linux systems that use 64 KiB pages
(`CONFIG_ARM64_64K_PAGES=y`), YJIT was panicking on boot due to a failed
assertion.
The assertion was making sure that code GC can free the last code page
that YJIT manages without freeing unrelated memory. YJIT prefers picking
16 KiB as the granularity at which to free code memory, but when the
system can only free at 64 KiB granularity, that is not possible.
The fix is to use the system page size as the code page size when the
system page size is 64 KiB. Continue to use 16 KiB as the code page size
on common systems that use 16/4 KiB pages.
Add asserts to code_gc() and free_page() about code GC's assumptions.
Fixes [Bug #19400]
---
yjit/src/asm/mod.rs | 78 ++++++++++++++++++++++++++++++++------------------
yjit/src/codegen.rs | 2 --
yjit/src/virtualmem.rs | 13 +++++++++
3 files changed, 63 insertions(+), 30 deletions(-)
YJIT: Fix assertion for partially mapped last pages (#7337)
Follows up [Bug #19400]
---
test/ruby/test_yjit.rb | 19 +++++++++++++++++++
yjit/src/asm/mod.rs | 2 +-
yjit/src/virtualmem.rs | 18 +++++++++++++-----
3 files changed, 33 insertions(+), 6 deletions(-)
Only emit circular dependency warning for owned thread shields [Bug
#19415]
If multiple threads attemps to load the same file concurrently
it's not a circular dependency issue.
So we check that the existing ThreadShield is owner by the current
fiber before warning about circular dependencies.
---
internal/thread.h | 1 +
load.c | 3 ++-
spec/ruby/core/kernel/shared/require.rb | 11 +++++++++++
spec/ruby/fixtures/code/concurrent_require_fixture.rb | 4 ++++
test/ruby/test_require.rb | 3 ---
thread.c | 11 +++++++++++
6 files changed, 29 insertions(+), 4 deletions(-)
create mode 100644 spec/ruby/fixtures/code/concurrent_require_fixture.rb
Use Thread.pass until thread.stop? to wait for thread to block
[Bug #19415]
It should be more reliable
---
spec/ruby/fixtures/code/concurrent_require_fixture.rb | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
mkconfig: Map `includedir` only for system ruby
Only when installing to the system path on macOS, prepend '$(SDKROOT)'
and remap `includedir`.
Fix https://github.com/rbenv/ruby-build/discussions/2123
---
test/mkmf/test_config.rb | 4 ++--
test/test_rbconfig.rb | 9 ---------
tool/mkconfig.rb | 4 +++-
3 files changed, 5 insertions(+), 12 deletions(-)
[Bug #19399] Parsing invalid heredoc inside block parameter
Although this is of course invalid as Ruby code, allow to just parse
and tokenize.
---
ext/ripper/lib/ripper/lexer.rb | 2 +-
test/ripper/test_lexer.rb | 12 ++++++++++++
2 files changed, 13 insertions(+), 1 deletion(-)
[Bug #19398] Memory leak in WeakMap
There's a memory leak in ObjectSpace::WeakMap due to not freeing
the `struct weakmap`. It can be seen in the following script:
```
100.times do
10000.times do
ObjectSpace::WeakMap.new
end
# Output the Resident Set Size (memory usage, in KB) of the current Ruby process
puts `ps -o rss= -p #{$$}`
end
```
---
gc.c | 1 +
test/ruby/test_weakmap.rb | 9 +++++++++
2 files changed, 10 insertions(+)
YJIT: Crash with rb_bug() when panicking
Helps with getting good bug reports in the wild. Intended to be
backported to the 3.2.x series.
---
yjit/bindgen/src/main.rs | 3 +++
yjit/src/cruby_bindings.inc.rs | 1 +
yjit/src/yjit.rs | 29 +++++++++++++++++++++++++++--
3 files changed, 31 insertions(+), 2 deletions(-)
[PATCH 1/4] YJIT: Move CodegenGlobals::freed_pages into an Rc
This allows for supplying a freed_pages vec in Rust tests. We need it so we
can test scenarios that occur after code GC.
---
yjit/src/asm/mod.rs | 48 +++++++++++++++++++++++++++++++++------------
yjit/src/codegen.rs | 16 ++++-----------
2 files changed, 39 insertions(+), 25 deletions(-)
Subject: [PATCH 2/4] YJIT: other_cb is None in tests
Since the other cb is in CodegenGlobals, and we want Rust tests to be
self-contained.
---
yjit/src/asm/mod.rs | 1 +
1 file changed, 1 insertion(+)
Subject: [PATCH 3/4] YJIT: ARM64: Move functions out of arm64_emit()
---
yjit/src/backend/arm64/mod.rs | 180 +++++++++++++++++-----------------
1 file changed, 90 insertions(+), 90 deletions(-)
Subject: [PATCH 4/4] YJIT: ARM64: Fix long jumps to labels
Previously, with Code GC, YJIT panicked while trying to emit a B.cond
instruction with an offset that is not encodable in 19 bits. This only
happens when the code in an assembler instance straddles two pages.
To fix this, when we detect that a jump to a label can land on a
different page, we switch to a fresh new page and regenerate all the
code in the assembler there. We still assume that no one assembler has
so much code that it wouldn't fit inside a fresh new page.
[Bug #19385]
---
yjit/src/backend/arm64/mod.rs | 65 ++++++++++++++++++++++++++++++++---
1 file changed, 60 insertions(+), 5 deletions(-)
Fix parsing of regexps that toggle extended mode on/off inside regexp
This was broken in ec3542229b. That commit
didn't handle cases where extended mode was turned on/off inside the
regexp. There are two ways to turn extended mode on/off:
```
/(?-x:#y)#z
/x =~ '#y'
/(?-x)#y(?x)#z
/x =~ '#y'
```
These can be nested inside the same regexp:
```
/(?-x:(?x)#x
(?-x)#y)#z
/x =~ '#y'
```
As you can probably imagine, this makes handling these regexps
somewhat complex. Due to the nesting inside portions of regexps,
the unassign_nonascii function needs to be recursive. In
recursive mode, it needs to track both opening and closing
parentheses, similar to how it already tracked opening and
closing brackets for character classes.
When scanning the regexp and coming to `(?` not followed by `#`,
scan for options, and use `x` and `i` to determine whether to
turn on or off extended mode. For `:`, indicting only the
current regexp section should have the extended mode
switched, recurse with the extended mode set or unset. For `)`,
indicating the remainder of the regexp (or current regexp portion
if already recursing) should turn extended mode on or off, just
change the extended mode flag and keep scanning.
While testing this, I noticed that `a`, `d`, and `u` are accepted
as options, in addition to `i`, `m`, and `x`, but I can't see
where those options are documented. I'm not sure whether or not
handling `a`, `d`, and `u` as options is a bug.
Fixes [Bug #19379]
---
re.c | 153 +++++++++++++++++++++++++++++++++++++----------
test/ruby/test_regexp.rb | 56 +++++++++++++++++
2 files changed, 176 insertions(+), 33 deletions(-)
bignum.c: rb_int_parse_cstr handle `0` strings
[Bug #19390]
We shouldn't check the string length when skipping zeros, as the
string might only contains zero characters, resulting in an empty string.
---
bignum.c | 1 -
test/ruby/test_time.rb | 4 ++++
2 files changed, 4 insertions(+), 1 deletion(-)
[Bug #19350]
* Merge RubyGems-3.4.2 and Bundler-2.4.2
* Merge RubyGems-3.4.3 and Bundler-2.4.3
* Generate parser-text.rb of racc when sync it
* Ignore LICENSE files of libraries vendored in rubygems [ci skip]
* Adjust spec of bundler like as `sync_default_gems` [ci skip]
* Fixed a typo
* Removed vendored LICENSE file.
* Update LEGAL sections for pub_grub
* Merge RubyGems-3.4.4 and Bundler-2.4.4
* Merge RubyGems-3.4.5 and Bundler-2.4.5
Co-authored-by: Nobuyoshi Nakada <nobu@ruby-lang.org>
[Bug #19289] Retain `ruby_abi_version` function
A few extension libraries, to hide all symbols except for necessary to
load, hardcode the symbols to be exported in symbol list files for
linker without even checking by `have_func`. As a workaround for such
libraries, retain `ruby_abi_version` symbol always even in released
versions for now.
---
include/ruby/internal/abi.h | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)
Avoid checking interrupt when loading iseq
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
The interrupt check will unintentionally release the VM lock when loading an iseq.
And this will cause issues with the `debug` gem's
[`ObjectSpace.each_iseq` method](0fcfc28aca/ext/debug/iseq_collector.c (L61-L67)),
which wraps iseqs with a wrapper and exposes their internal states when they're actually not ready to be used.
And when that happens, errors like this would occur and kill the `debug` gem's thread:
```
DEBUGGER: ReaderThreadError: uninitialized InstructionSequence
┃ DEBUGGER: Disconnected.
┃ ["/opt/rubies/ruby-3.2.0/lib/ruby/gems/3.2.0/gems/debug-1.7.1/lib/debug/breakpoint.rb:247:in `absolute_path'",
┃ "/opt/rubies/ruby-3.2.0/lib/ruby/gems/3.2.0/gems/debug-1.7.1/lib/debug/breakpoint.rb:247:in `block in iterate_iseq'",
┃ "/opt/rubies/ruby-3.2.0/lib/ruby/gems/3.2.0/gems/debug-1.7.1/lib/debug/breakpoint.rb:246:in `each_iseq'",
...
```
A way to reproduce the issue is to satisfy these conditions at the same time:
1. `debug` gem calling `ObjectSpace.each_iseq` (e.g. [activating a `LineBreakpoint`](0fcfc28aca/lib/debug/breakpoint.rb (L246))).
2. A large amount of iseq being loaded from another thread (possibly through the `bootsnap` gem).
3. 1 and 2 iterating through the same iseq(s) at the same time.
Because this issue requires external dependencies and a rather complicated timing setup to reproduce, I wasn't able to write a test case for it.
But here's some pseudo code to help reproduce it:
```rb
require "debug/session"
Thread.new do
100.times do
ObjectSpace.each_iseq do |iseq|
iseq.absolute_path
end
end
end
sleep 0.1
load_a_bunch_of_iseq
possibly_through_bootsnap
```
[Bug #19348]
Co-authored-by: Peter Zhu <peter@peterzhu.ca>
---
compile.c | 2 +-
vm_core.h | 1 +
vm_insnhelper.c | 11 +++++++++++
3 files changed, 13 insertions(+), 1 deletion(-)
Fix off-by-one error in rb_vm_each_stack_value
Applying the following patch to test/erb/test_erb.rb and running that
file will cause Ruby to crash on my machine (macOS 13.1 on M1 Pro):
```
--- a/test/erb/test_erb.rb
+++ b/test/erb/test_erb.rb
@@ -7,6 +7,12 @@
class TestERB < Test::Unit::TestCase
class MyError < RuntimeError ; end
+ def setup
+ GC.auto_compact = true
+ GC.stress = true
+ GC.verify_compaction_references(expand_heap: true, toward: :empty)
+ end
+
```
It crashes with the following log:
```
/Users/peter/src/ruby/lib/erb/compiler.rb:276: [BUG] Segmentation fault at 0x00000001083a8690
...
-- C level backtrace information -------------------------------------------
...
/Users/peter/src/ruby/build/ruby(rb_vm_each_stack_value+0xa8) [0x104cc3a44] ../vm.c:2737
/Users/peter/src/ruby/build/ruby(rb_vm_each_stack_value+0xa8) [0x104cc3a44] ../vm.c:2737
/Users/peter/src/ruby/build/ruby(check_stack_for_moved+0x2c) [0x104b272a4] ../gc.c:5512
/Users/peter/src/ruby/build/ruby(gc_compact_finish) ../gc.c:5534
/Users/peter/src/ruby/build/ruby(gc_sweep_compact) ../gc.c:8653
/Users/peter/src/ruby/build/ruby(gc_sweep) ../gc.c:6196
/Users/peter/src/ruby/build/ruby(has_sweeping_pages+0x0) [0x104b19c54] ../gc.c:9568
/Users/peter/src/ruby/build/ruby(gc_rest) ../gc.c:9570
```
This crash happens because it's reading the VALUE at sp. But since
sp points to the top of the stack, it's reading the VALUE above the
top of the stack, which is causing this segfault.
Fixes [Bug #19320]
---
vm.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
String#bytesplice should return self
In Feature #19314, we concluded that the return value of String#bytesplice
should be changed from the source string to the receiver, because the source
string is useless and confusing when extra arguments are added.
This change should be included in Ruby 3.2.1.
---
string.c | 4 ++--
test/ruby/test_string.rb | 2 +-
2 files changed, 3 insertions(+), 3 deletions(-)
Fix re-embedding of strings during compaction
The reference updating code for strings is not re-embedding strings
because the code is incorrectly wrapped inside of a
`if (STR_SHARED_P(obj))` clause. Shared strings can't be re-embedded
so this ends up being a no-op. This means that strings can be moved to a
large size pool during compaction, but won't be re-embedded, which would
waste the space.
---
gc.c | 16 +++++++++-------
string.c | 12 ++++++++----
test/ruby/test_gc_compact.rb | 8 ++++----
3 files changed, 21 insertions(+), 15 deletions(-)
[Bug #19319] Fix crash in rb_str_casemap
The following code crashes on my machine:
```
GC.stress = true
str = "testing testing testing"
puts str.capitalize
```
We need to ensure that the object `buffer_anchor` remains on the stack
so it does not get GC'd.
---
string.c | 2 ++
1 file changed, 2 insertions(+)
Fix undefined behavior in shape.c
Under strict aliasing, writing to the memory location of a different
type is not allowed and will result in undefined behavior. This was
happening in shape.c due to `rb_id_table_lookup` writing to the memory
location of `VALUE *` that was casted from a `rb_shape_t **`.
This was causing test failures when compiled with LTO.
Fixes [Bug #19248]
Co-Authored-By: Alan Wu <alanwu@ruby-lang.org>
---
shape.c | 13 +++++++++++--
1 file changed, 11 insertions(+), 2 deletions(-)
Fix crash in TracePoint c_call for removed method
trace_arg->id is the ID of the original method of an aliased method. If
the original method is removed, then the lookup will fail. We should use
trace_arg->called_id instead, which is the ID of the aliased method.
Fixes [Bug #19305]
---
test/ruby/test_settracefunc.rb | 23 +++++++++++++++++++++++
vm_trace.c | 2 +-
2 files changed, 24 insertions(+), 1 deletion(-)
YJIT: Save PC and SP before calling leaf builtins (#7090)
Previously, we did not update `cfp->sp` before calling the C function of
ISEQs marked with `Primitive.attr! "inline"` (leaf builtins). This
caused the GC to miss temporary values on the stack in case the function
allocates and triggers a GC run. Right now, there is only a few leaf
builtins in numeric.rb on Integer methods such as `Integer#~`. Since
these methods only allocate when operating on big numbers, we missed
this issue.
Fix by saving PC and SP before calling the functions -- our usual
protocol for calling C functions that may allocate on the GC heap.
[Bug #19316]
---
test/ruby/test_yjit.rb | 16 ++++++++++++++++
yjit/src/codegen.rs | 4 ++++
2 files changed, 20 insertions(+)
common.mk: Do not invoke outdate-bundled-gems by default
If baseruby is available (and its version is different from one being
built) when compiling ruby, tool/outdate-bundled-gems.rb (which is
invoked by `make install`) wrongly deletes debug.so and rbs_extension.so
in .bundle/extension/*.
This leads to a broken installation of ruby which lacks the libraries,
which may make rubygems show the following warnings (in some additional
complex conditions):
```
$ irb
Ignoring debug-1.7.1 because its extensions are not built. Try: gem pristine debug --version 1.7.1
Ignoring rbs-2.8.2 because its extensions are not built. Try: gem pristine rbs --version 2.8.2
irb(main):001:0>
```
According to some committers, tool/outdate-bundled-gems.rb is introduced
for fixing a build issue, but the detail is not recorded. The issue
seems to occur only when debug gem or rbs gem is updated, so it is
difficult to fix the script so soon.
Tentatively, this change stops invoking the script by default.
This should be backported to ruby_3_2.
Fixes [Bug #19271]
---
common.mk | 1 -
1 file changed, 1 deletion(-)
YJIT: Fix `yield` into block with >=30 locals on ARM
It's a register spill issue. Fix by moving the Qnil fill snippet to
after registers are released.
[Bug #19299]
---
test/ruby/test_yjit.rb | 14 ++++++++++++++
yjit/src/codegen.rs | 29 ++++++++++++++---------------
2 files changed, 28 insertions(+), 15 deletions(-)
Return 0 if there is no CFP on the EC yet
StackProf uses a signal handler to call `rb_profile_frames`. Signals
are delivered to threads randomly, and can be delivered after the thread
has been created but before the CFP has been established on the EC.
This commit returns early if there is no CFP to use.
Here is some info from the core files we are seeing. Here you can see
the CFP on the current EC is 0x0:
```
(gdb) p ruby_current_ec
$20 = (struct rb_execution_context_struct *) 0x7f3481301b50
(gdb) p ruby_current_ec->cfp
$21 = (rb_control_frame_t *) 0x0
```
Here is where VM_FRAME_CFRAME_P gets a 0x0 CFP:
```
6 VM_FRAME_CFRAME_P (cfp=0x0) at vm_core.h:1350
7 VM_FRAME_RUBYFRAME_P (cfp=<optimized out>) at vm_core.h:1350
8 rb_profile_frames (start=0, limit=2048, buff=0x7f3493809590, lines=0x7f349380d590) at vm_backtrace.c:1587
```
Down the stack we can see this is happening after thread creation:
```
19 0x00007f3495bf9420 in <signal handler called> () at /lib/x86_64-linux-gnu/libpthread.so.0
20 0x000055d531574e55 in thread_start_func_2 (th=<optimized out>, stack_start=<optimized out>) at thread.c:676
21 0x000055d531575b31 in thread_start_func_1 (th_ptr=<optimized out>) at thread_pthread.c:1170
22 0x00007f3495bed609 in start_thread (arg=<optimized out>) at pthread_create.c:477
23 0x00007f3495b12133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
```
---
vm_backtrace.c | 9 +++++++++
1 file changed, 9 insertions(+)
Fix integer underflow when using HEAP_INIT_SLOTS
There is an integer underflow when the environment variable
RUBY_GC_HEAP_INIT_SLOTS is less than the number of slots currently
in the Ruby heap.
[Bug #19284]
---
gc.c | 25 +++++++++++++------------
test/ruby/test_gc.rb | 5 +++++
2 files changed, 18 insertions(+), 12 deletions(-)
NARUSE, Yui
Hiroshi SHIBATA
Peter Zhu
Kazuhiro NISHIYAMA
Nobuyoshi Nakada
git
David Rodríguez