The following new debug context APIs are for implementing debugger's
`next` (step over) and similar functionality.
* `rb_debug_inspector_frame_depth(dc, index)` returns `index`-th
frame's depth.
* `rb_debug_inspector_current_depth()` returns current frame depth.
The frame depth is not related to the frame index because debug
context API skips some special frames but proposed `_depth()` APIs
returns the count of all frames (raw depth).
Since https://github.com/ruby/ruby/pull/6599, RUBY_IMMEDIATE_MASK also
overlaps RUBY_Qnil. Now RB_SPECIAL_CONST_P seems confusing since both
RB_IMMEDIATE_P and RB_TEST check for RUBY_Qnil while we only need to
check RUBY_Qnil besides RUBY_IMMEDIATE_MASK. I'd like to make this
change to make it less confusing.
I confirmed that this doesn't change the number of instructions used for
the RUBY_Qfalse check on Linux x86_64 GCC and macOS arm64 Clang.
* Windows: Fix warning about undefined if_indextoname()
* Windows: Fix UNIXSocket on MINGW and make .pair more reliable
* Windows: Use nonblock=true for read tests with scheduler
* Windows: Move socket detection from File.socket? to File.stat
Add S_IFSOCK to Windows and interpret reparse points accordingly.
Enable tests that work now.
* Windows: Use wide-char functions to UNIXSocket
This fixes behaviour with non-ASCII characters.
It also fixes deletion of temporary UNIXSocket.pair files.
* Windows: Add UNIXSocket tests for specifics of Windows impl.
* Windows: fix VC build due to missing _snwprintf
Avoid usage of _snwprintf, since it fails linking ruby.dll like so:
linking shared-library x64-vcruntime140-ruby320.dll
x64-vcruntime140-ruby320.def : error LNK2001: unresolved external symbol snwprintf
x64-vcruntime140-ruby320.def : error LNK2001: unresolved external symbol vsnwprintf_l
whereas linking miniruby.exe succeeds.
This patch uses snprintf on the UTF-8 string instead.
Also remove branch GetWindowsDirectoryW, since it doesn't work.
* Windows: Fix dangling symlink test failures
Co-authored-by: Lars Kanis <kanis@comcard.de>
* Windows: Use readlink emulation for File.readlink
This fixes readlink emulation for the ERROR_MORE_DATA case and general error reporting.
It now releases GVL while readlink IO operation.
The dedicated rb_readlink was introduced in commit 2ffb87995a
in order to improve encoding and buffer allocation.
However the encoding issues are solved since ruby-3.0 switched to UTF-8
and the buffer allocation will be improved in a later commit.
* Windows: Increase the default buffer size for reparse point info
So far nearly all queries of reparse points needed two attempts to get enough buffer.
* Windows: Remove declaration of rb_w32_wreadlink
It was removed in commit 2f6fdd3aeb
Since object shapes store the capacity of an object, we no longer
need the numiv field on RObjects. This gives us one extra slot which
we can use to give embedded objects one more instance variable (for a
total of 3 ivs). This commit removes the concept of numiv from RObject.
This commit adds a `capacity` field to shapes, and adds shape
transitions whenever an object's capacity changes. Objects which are
allocated out of a bigger size pool will also make a transition from the
root shape to the shape with the correct capacity for their size pool
when they are allocated.
This commit will allow us to remove numiv from objects completely, and
will also mean we can guarantee that if two objects share shapes, their
IVs are in the same positions (an embedded and extended object cannot
share shapes). This will enable us to implement ivar sets in YJIT using
object shapes.
Co-Authored-By: Aaron Patterson <tenderlove@ruby-lang.org>
A code pattern `p + enclen(enc, p, pend)` may lead to a buffer overrun
if incomplete bytes of a UTF-8 character is placed at the end of a
string. Because this pattern is used in several places in onigmo,
this change fixes the issue in the side of `enclen`: the function should
not return a number that is larger than `pend - p`.
Co-Authored-By: Nobuyoshi Nakada <nobu@ruby-lang.org>
Implements [Feature #12084]
Returns the object for which the receiver is the singleton class, or
raises TypeError if the receiver is not a singleton class.
Object Shapes is used for accessing instance variables and representing the
"frozenness" of objects. Object instances have a "shape" and the shape
represents some attributes of the object (currently which instance variables are
set and the "frozenness"). Shapes form a tree data structure, and when a new
instance variable is set on an object, that object "transitions" to a new shape
in the shape tree. Each shape has an ID that is used for caching. The shape
structure is independent of class, so objects of different types can have the
same shape.
For example:
```ruby
class Foo
def initialize
# Starts with shape id 0
@a = 1 # transitions to shape id 1
@b = 1 # transitions to shape id 2
end
end
class Bar
def initialize
# Starts with shape id 0
@a = 1 # transitions to shape id 1
@b = 1 # transitions to shape id 2
end
end
foo = Foo.new # `foo` has shape id 2
bar = Bar.new # `bar` has shape id 2
```
Both `foo` and `bar` instances have the same shape because they both set
instance variables of the same name in the same order.
This technique can help to improve inline cache hits as well as generate more
efficient machine code in JIT compilers.
This commit also adds some methods for debugging shapes on objects. See
`RubyVM::Shape` for more details.
For more context on Object Shapes, see [Feature: #18776]
Co-Authored-By: Aaron Patterson <tenderlove@ruby-lang.org>
Co-Authored-By: Eileen M. Uchitelle <eileencodes@gmail.com>
Co-Authored-By: John Hawthorn <john@hawthorn.email>
Object Shapes is used for accessing instance variables and representing the
"frozenness" of objects. Object instances have a "shape" and the shape
represents some attributes of the object (currently which instance variables are
set and the "frozenness"). Shapes form a tree data structure, and when a new
instance variable is set on an object, that object "transitions" to a new shape
in the shape tree. Each shape has an ID that is used for caching. The shape
structure is independent of class, so objects of different types can have the
same shape.
For example:
```ruby
class Foo
def initialize
# Starts with shape id 0
@a = 1 # transitions to shape id 1
@b = 1 # transitions to shape id 2
end
end
class Bar
def initialize
# Starts with shape id 0
@a = 1 # transitions to shape id 1
@b = 1 # transitions to shape id 2
end
end
foo = Foo.new # `foo` has shape id 2
bar = Bar.new # `bar` has shape id 2
```
Both `foo` and `bar` instances have the same shape because they both set
instance variables of the same name in the same order.
This technique can help to improve inline cache hits as well as generate more
efficient machine code in JIT compilers.
This commit also adds some methods for debugging shapes on objects. See
`RubyVM::Shape` for more details.
For more context on Object Shapes, see [Feature: #18776]
Co-Authored-By: Aaron Patterson <tenderlove@ruby-lang.org>
Co-Authored-By: Eileen M. Uchitelle <eileencodes@gmail.com>
Co-Authored-By: John Hawthorn <john@hawthorn.email>
Previously, newline: :lf was accepted but ignored. Where it
should have been used was commented out code that didn't work,
but unlike all other invalid values, using newline: :lf did
not raise an error.
This adds support for newline: :lf and :lf_newline, for consistency
with newline: :cr and :cr_newline. This is basically the same as
universal_newline, except that it only affects writing and not
reading due to RUBY_ECONV_NEWLINE_DECORATOR_WRITE_MASK.
Add tests for the File.open :newline option while here.
Fixes [Bug #12436]
As commented in include/ruby/internal/abi.h, since teeny versions of
Ruby should guarantee ABI compatibility, `RUBY_ABI_VERSION` has no role
in released versions of Ruby.
rb_ary_tmp_new suggests that the array is temporary in some way, but
that's not true, it just creates an array that's hidden and not on the
transient heap. This commit renames it to rb_ary_hidden_new.
This commit implements Objects on Variable Width Allocation. This allows
Objects with more ivars to be embedded (i.e. contents directly follow the
object header) which improves performance through better cache locality.
The (sole) use of memcpy in our public header is now replaced to
directly call ruby_nonempty_memcpy, and the previous definition of
memcpy is now internal-only. [Bug#18893]
Since enabling YJIT or MJIT drastically changes what could go wrong at
runtime, it's good to be front and center about whether they are enabled
when dumping a crash report. Previously, `RUBY_DESCRIPTION` and the
description printed when crashing can be different when a JIT is on.
Introduce a new internal data global, `rb_dynamic_description`, and set
it to be the same as `RUBY_DESCRIPTION` during initialization; use it
when crashing.
* version.c: Init_ruby_description(): Initialize and use
`rb_dynamic_description`.
* error.c: Change crash reports to use `rb_dynamic_description`.
* ruby.c: Call `Init_ruby_description()` earlier. Slightly more work
for when we exit right after printing the description but that
was deemed acceptable.
* include/ruby/version.h: Talk about how JIT info is not in
`ruby_description`.
* test/-ext-/bug_reporter/test_bug_reporter.rb: Remove handling for
crash description being different from `RUBY_DESCRIPTION`.
* test/ruby/test_rubyoptions.rb: ditto
Co-authored-by: Nobuyoshi Nakada <nobu@ruby-lang.org>
Co-authored-by: Alan Wu <alanwu@ruby-lang.org>
[Feature #18339]
After experimenting with the initial version of the API I figured there is a need
for an exit event to cleanup instrumentation data. e.g. if you record data in a
{thread_id -> data} table, you need to free associated data when a thread goes away.
This function was added to a public header in [1] probably
unintentionally since it's not used anywhere, exposes implementation
details, and isn't related to the goals of that pull request.
[1]: 56cc3e99b6
Previously, because opt_aref and opt_aset don't push a frame, when they
would call rb_hash to determine the hash value of the key, the initial
level of recursion would incorrectly use the method id at the top of the
stack instead of "hash".
This commit replaces rb_exec_recursive_outer with
rb_exec_recursive_outer_mid, which takes an explicit method id, so that
we can make the hash calculation behave consistently.
rb_exec_recursive_outer was documented as being internal, so I believe
this should be okay to change.
Ref: https://bugs.ruby-lang.org/issues/18339
Design:
- This tries to minimize the overhead when no hook is registered.
It should only incur an extra unsynchronized boolean check.
- The hook list is protected with a read-write lock as to cause
contention when some hooks are registered.
- The hooks MUST be thread safe, and MUST NOT call into Ruby as they
are executed outside the GVL.
- It's simply a noop on Windows.
API:
```
rb_internal_thread_event_hook_t * rb_internal_thread_add_event_hook(rb_internal_thread_event_callback callback, rb_event_flag_t internal_event, void *user_data);
bool rb_internal_thread_remove_event_hook(rb_internal_thread_event_hook_t * hook);
```
You can subscribe to 3 events:
- READY: called right before attempting to acquire the GVL
- RESUMED: called right after successfully acquiring the GVL
- SUSPENDED: called right after releasing the GVL.
The hooks MUST be threadsafe, as they are executed outside of the GVL, they also MUST NOT call any Ruby API.
Having more size pools will allow us to allocate larger objects
through Variable Width Allocation.
I have attached some benchmark results below.
Discourse:
On Discourse, we don't see much change in response times. We do see
a small reduction in RSS.
Branch RSS: 377.8 MB
Master RSS: 396.3 MB
railsbench:
On railsbench, we don't see a big change in RPS or p99 performance.
We see a small increase in RSS.
Branch RPS: 815.38
Master RPS: 811.73
Branch p99: 1.69 ms
Master p99: 1.68 ms
Branch RSS: 90.6 MB
Master RSS: 89.4 MB
liquid:
We don't see a significant change in liquid performance.
Branch parse & render: 29.041 I/s
Master parse & render: 29.211 I/s
[Feature #18683]
This allows parsers and similar libraries to create Hashes of
a certain capacity in advance. It's useful when the key and values
are streamed, hence `bulk_insert()` can't be used.
This commit reintroduces finer-grained constant cache invalidation.
After 8008fb7 got merged, it was causing issues on token-threaded
builds (such as on Windows).
The issue was that when you're iterating through instruction sequences
and using the translator functions to get back the instruction structs,
you're either using `rb_vm_insn_null_translator` or
`rb_vm_insn_addr2insn2` depending if it's a direct-threading build.
`rb_vm_insn_addr2insn2` does some normalization to always return to
you the non-trace version of whatever instruction you're looking at.
`rb_vm_insn_null_translator` does not do that normalization.
This means that when you're looping through the instructions if you're
trying to do an opcode comparison, it can change depending on the type
of threading that you're using. This can be very confusing. So, this
commit creates a new translator function
`rb_vm_insn_normalizing_translator` to always return the non-trace
version so that opcode comparisons don't have to worry about different
configurations.
[Feature #18589]
This reverts commits for [Feature #18589]:
* 8008fb7352
"Update formatting per feedback"
* 8f6eaca2e1
"Delete ID from constant cache table if it becomes empty on ISEQ free"
* 629908586b
"Finer-grained inline constant cache invalidation"
MSWin builds on AppVeyor have been crashing since the merger.
Current behavior - caches depend on a global counter. All constant mutations cause caches to be invalidated.
```ruby
class A
B = 1
end
def foo
A::B # inline cache depends on global counter
end
foo # populate inline cache
foo # hit inline cache
C = 1 # global counter increments, all caches are invalidated
foo # misses inline cache due to `C = 1`
```
Proposed behavior - caches depend on name components. Only constant mutations with corresponding names will invalidate the cache.
```ruby
class A
B = 1
end
def foo
A::B # inline cache depends constants named "A" and "B"
end
foo # populate inline cache
foo # hit inline cache
C = 1 # caches that depend on the name "C" are invalidated
foo # hits inline cache because IC only depends on "A" and "B"
```
Examples of breaking the new cache:
```ruby
module C
# Breaks `foo` cache because "A" constant is set and the cache in foo depends
# on "A" and "B"
class A; end
end
B = 1
```
We expect the new cache scheme to be invalidated less often because names aren't frequently reused. With the cache being invalidated less, we can rely on its stability more to keep our constant references fast and reduce the need to throw away generated code in YJIT.
This commit implements arrays on Variable Width Allocation. This allows
longer arrays to be embedded (i.e. contents directly follow the object
header) which improves performance through better cache locality.
Header file include/ruby/internal/abi.h contains RUBY_ABI_VERSION which
is the ABI version. This value should be bumped whenever an ABI
incompatible change is introduced.
When loading dynamic libraries, Ruby will compare its own
`ruby_abi_version` and the `ruby_abi_version` of the loaded library. If
these two values don't match it will raise a `LoadError`. This feature
can also be turned off by setting the environment variable
`RUBY_RUBY_ABI_CHECK=0`.
This feature will prevent cases where previously installed native gems
fail in unexpected ways due to incompatibility of changes in header
files. This will force the developer to recompile their gems to use the
same header files as the built Ruby.
In Ruby, the ABI version is exposed through
`RbConfig::CONFIG["ruby_abi_version"]`.
Configuration for mingw32 can't detect 'shutdown' due to wrong -l
option even though it's available (this has been going on for a while,
and it needs to be fixed).
In this situation, include/ruby/missing.h declares a stub shutdown
function since 7ee786388a, and another shutdown decl is came from
system header. They are incompatible at stdcall attribute, so it
causes compilation failure.
This change defines a HAVE_SHUTDOWN to guard a newly introduced stub
decl in include/ruby/missing.h
At least OpenBSD/sparc64 doesn't appear to define them, and possibly
some other OpenBSD GCC platforms don't (most OpenBSD platforms have
already switched to clang).
Nobuyoshi Nakada
Yukihiro "Matz" Matsumoto
Benoit Daloze
TSUYUSATO Kitsune
Matt Valentine-House
Jimmy Bourassa
卜部昌平
Lars Kanis
Samuel Williams
Koichi Sasada
Takashi Kokubun
Lars Kanis
Jemma Issroff
Yusuke Endoh
Nick Hengeveld
Ufuk Kayserilioglu
Sergey Fedorov
Jean Boussier
Aaron Patterson
spaette
Jeremy Evans
Peter Zhu
Kevin Backhouse
Chris Seaton
Jean Boussier
Alan Wu
John Hawthorn
Daisuke Fujimura (fd0)
Alex Matchneer
Kevin Newton
Yuta Saito