Support invokesuper in a block on YJIT
invokesuper previously side exited when it is in a block. To make sure we're compiling the correct method in super, we now use the local environment pointer (LEP) to get the method, which will work in a block.
Co-authored-by: John Hawthorn <john@hawthorn.email>
YJIT: Implement splat for cfuncs. Split exit cases
This also implements a new check for ruby2keywords as the last
argument of a splat. This does mean that we generate more code, but in
actual benchmarks where we gained speed from this (binarytrees) I
don't see any significant slow down. I did have to struggle here with
the register allocator to find code that didn't allocate too many
registers. It's a bit hard when everything is implicit. But I think I
got to the minimal amount of copying and stuff given our current
allocation strategy.
I noticed this while running test_yjit with --mjit-call-threshold=1,
which redefines `Integer#<`. When Ruby is monkey-patched,
MJIT itself could be broken.
Similarly, Ruby scripts could break MJIT in many different ways. I
prepared the same set of hooks as YJIT so that we could possibly
override it and disable it on those moments. Every constant under
RubyVM::MJIT is private and thus it's an unsupported behavior though.
* YJIT: Make case-when optimization respect === redefinition
Even when a fixnum key is in the dispatch hash, if there is a case such
that its basic operations for === is redefined, we need to fall back to
checking each case like the interpreter. Semantically we're always
checking each case by calling === in order, it's just that this is not
observable when basic operations are intact.
When all the keys are fixnums, though, we can do the optimization we're
doing right now. Check for this condition.
* Update yjit/src/cruby_bindings.inc.rs
Co-authored-by: Takashi Kokubun <takashikkbn@gmail.com>
Co-authored-by: Takashi Kokubun <takashikkbn@gmail.com>
Previously we essentially never freed block even after invalidation.
Their reference count never reached zero for a couple of reasons:
1. `Branch::block` formed a cycle with the block holding the branch
2. Strong count on a branch that has ever contained a stub never
reached 0 because we increment the `.clone()` call for
`BranchRef::into_raw()` didn't have a matching decrement.
It's not safe to immediately deallocate blocks during
invalidation since `branch_stub_hit()` can end up
running with a branch pointer from an invalidated branch.
To plug the leaks, we wait until code GC or global invalidation and
deallocate the blocks for iseqs that are definitely not running.
@casperisfine reporting a bug in this gist https://gist.github.com/casperisfine/d59e297fba38eb3905a3d7152b9e9350
After investigating I found it was caused by a combination of send and a c_func that we have overwritten in the JIT. For send calls, we need to do some stack manipulation before making the call. Because of the way exits works, we need to do that stack manipulation at the last possible moment. In this case, we weren't doing that stack manipulation at all. Unfortunately, with how the code is structured there isn't a great place to do that stack manipulation for our overridden C funcs.
Each overridden C func can return a boolean stating that it shouldn't be used. We would need to do the stack manipulation after all of those checks are done. We could pass a lambda(?) or separate out the logic for "can I run this override" from "now generate the code for it". Since we are coming up on a release, I went with the path of least resistence and just decided to not use these overrides if we are in a send call.
We definitely should revist this in the future.
Follow-up for 2b8191bdad. Since that
commit, we stopped doing code invalidation the second time the call and
return events are enabled. We need to do it every time these events are
enabled because we might have generated code while these events are
disabled.
Also rename locals and edit comments to make it more clear that the iseq
rewrite code path only happens the first time a particular iseq trace
event is enabled.
* YJIT: fix a parameter name
* YJIT: add support for calling bmethods
This commit adds support for the VM_METHOD_TYPE_BMETHOD method type in
YJIT. You can get these type of methods from facilities like
Kernel#define_singleton_method and Module#define_method.
Even though the body of these methods are blocks, the parameter setup
for them is exactly the same as VM_METHOD_TYPE_ISEQ, so we can reuse
the same logic in gen_send_iseq(). You can see this from how
vm_call_bmethod() eventually calls setup_parameters_complex() with
arg_setup_method.
Bmethods do need their frame environment to be setup differently. We
handle this by allowing callers of gen_send_iseq() to control the iseq,
the frame flag, and the prev_ep. The `prev_ep` goes into the same
location as the block handler would go into in an iseq method frame.
Co-authored-by: John Hawthorn <john@hawthorn.email>
Co-authored-by: John Hawthorn <john@hawthorn.email>
* Create code generation func
* Make rb_vm_concat_array available to use in Rust
* Map opcode to code gen func
* Implement code gen for concatarray
* Add test for concatarray
* Use new asm backend
* Add comment to C func wrapper
`YJIT.simulate_oom!` used to leave one byte of space in the code block,
so our test didn't expose a problem with asserting that the write
position is in bounds in `CodeBlock::set_pos`. We do the following when
patching code:
1. save current write position
2. seek to middle of the code block and patch
3. restore old write position
The bounds check fails on (3) when the code block is already filled up.
Leaving one byte of space also meant that when we write that byte, we
need to fill the entire code region with trapping instruction in
`VirtualMem`, which made the OOM tests unnecessarily slow.
Remove the incorrect bounds check and stop leaving space in the code
block when simulating OOM.
* Port gen_send_iseq to the new backend IR
* Replace occurrences of 8 by SIZEOF_VALUE
Co-authored-by: Maxime Chevalier-Boisvert <maximechevalierb@gmail.com>
Co-authored-by: Maxime Chevalier-Boisvert <maxime.chevalierboisvert@shopify.com>
* Port gen_send_cfunc to the new backend
* Remove an obsoleted test
* Add more cfunc tests
* Use csel_e instead and more into()
Co-authored-by: Maxime Chevalier-Boisvert <maximechevalierb@gmail.com>
* Add a missing lea for build_kwargs
* Split cfunc test cases
Co-authored-by: Maxime Chevalier-Boisvert <maximechevalierb@gmail.com>
Teach getblockparamproxy to handle the no-block case without exiting
Co-authored-by: John Hawthorn <john@hawthorn.email>
Co-authored-by: John Hawthorn <john@hawthorn.email>
The test in [1] was removed because it stopped working when we limited
the power of Kernel#binding in [2]. However, the underlying issue could
still be reproduced using blocks. Add back a regression test.
I tested the test by commenting out the fix from [1].
[1]: 54c91042ed
[2]: 343ea9967e
In December 2021, we opened an [issue] to solicit feedback regarding the
porting of the YJIT codebase from C99 to Rust. There were some
reservations, but this project was given the go ahead by Ruby core
developers and Matz. Since then, we have successfully completed the port
of YJIT to Rust.
The new Rust version of YJIT has reached parity with the C version, in
that it passes all the CRuby tests, is able to run all of the YJIT
benchmarks, and performs similarly to the C version (because it works
the same way and largely generates the same machine code). We've even
incorporated some design improvements, such as a more fine-grained
constant invalidation mechanism which we expect will make a big
difference in Ruby on Rails applications.
Because we want to be careful, YJIT is guarded behind a configure
option:
```shell
./configure --enable-yjit # Build YJIT in release mode
./configure --enable-yjit=dev # Build YJIT in dev/debug mode
```
By default, YJIT does not get compiled and cargo/rustc is not required.
If YJIT is built in dev mode, then `cargo` is used to fetch development
dependencies, but when building in release, `cargo` is not required,
only `rustc`. At the moment YJIT requires Rust 1.60.0 or newer.
The YJIT command-line options remain mostly unchanged, and more details
about the build process are documented in `doc/yjit/yjit.md`.
The CI tests have been updated and do not take any more resources than
before.
The development history of the Rust port is available at the following
commit for interested parties:
1fd9573d8b
Our hope is that Rust YJIT will be compiled and included as a part of
system packages and compiled binaries of the Ruby 3.2 release. We do not
anticipate any major problems as Rust is well supported on every
platform which YJIT supports, but to make sure that this process works
smoothly, we would like to reach out to those who take care of building
systems packages before the 3.2 release is shipped and resolve any
issues that may come up.
[issue]: https://bugs.ruby-lang.org/issues/18481
Co-authored-by: Maxime Chevalier-Boisvert <maximechevalierb@gmail.com>
Co-authored-by: Noah Gibbs <the.codefolio.guy@gmail.com>
Co-authored-by: Kevin Newton <kddnewton@gmail.com>
Check whether the current or previous frame is a Ruby frame in
call_trace_func and rb_tracearg_binding before attempting to
create a binding for the frame.
Fixes [Bug #18487]
Co-authored-by: Alan Wu <XrXr@users.noreply.github.com>
Check whether the current or previous frame is a Ruby frame in
call_trace_func before attempting to create a binding for the frame.
Fixes [Bug #18487]
Co-authored-by: Alan Wu <XrXr@users.noreply.github.com>
This adds support for passing keyword arguments to cfuncs. This is done
by calling a helper method to create the hash from the top N values on
the stack (determined by the callinfo) and then moving that value onto
the stack.
Routines that are called from YJIT's output code can call methods, and
calling methods mean they can capture and change the environment of the
calling frame.
Discard type info whenever we perform routine calls. This is more
conservative than strictly necessary as some routines need to perform GC
allocation but can never call methods and so should never be able to
change local variables. However, manually analyzing C functions for
whether they have code paths that call methods is error prone and can go
out of date as changes land in the codebase.
Closes: shopify/yjit#300
Previously when we were calling a method with an optional argument and
multiple keywords arguments which weren't in the order the receiver
expected we would use the wrong SP index to rearrange them.
Fixes Bug #18453
Previously, YJIT would not check that all the required keywords were
specified in the case that there were optional arguments specified. In
this case YJIT would incorrectly call the method with invalid arguments.
Previously we mirrored the fast paths the interpreter had for having
only one of kwargs or optional args. This commit aims to combine the
cases and reduce complexity.
Though this allows calling iseqs which have have both optional and
keyword arguments, it requires that all optional arguments are specified
when there are keyword arguments, since unspecified optional arguments
appear before the kwargs. Support for this can be added a in a future
PR.
Previously, YJIT incorrectly discarded the upper 32 bits of the object
pointer when writing out VALUEs to setup default keyword arguments.
In addition to incorrectly truncating, the output pointers were not
properly tracked for handling GC compaction moving the referenced
objects.
YJIT previously attempted to encode a mov instruction with a memory
destination and a 64 bit immediate when there is no such encoding
possible in the ISA. Add an assert to mitigate not being able to
catch this at build time.
It's superseded by functionality added to jit_guard_known_klass().
In weird situations such as the ones in the included test,
guard_self_is_heap() triggered assertions.
Co-authored-by: Jemma Issroff <jemmaissroff@gmail.com>
As part of YJIT's strategy for promoting Ruby constant expressions into
constants in the output native code, the interpreter calls
rb_yjit_constant_ic_update() from opt_setinlinecache.
The block invalidation loop indirectly calls rb_darray_remove_unordered(),
which does a shuffle remove. Because of this, looping with an
incrementing counter like done previously can miss some elements in the
array. Repeatedly invalidate the first element instead.
The bug this commit resolves does not seem to cause crashes or divergent
behaviors.
Co-authored-by: Jemma Issroff <jemmaissroff@gmail.com>
As of [1] and [2], YJIT has enough support for out of memory conditions
to pass these two basic tests.
OOM code paths are prone to bugs since they are rarely exercised in
common workloads. We might want to add CI runs that stress test these
code paths. Maybe outside of GitHub Actions for capacity reasons.
[1]: f41b4d44f9
[2]: b5b6ab4194
Previously, YJIT crashes with rb_bug() when asked to compile new methods
while out of executable memory.
To handle this situation gracefully, this change keeps track of all the
blocks compiled each invocation in case YJIT runs out of memory in the
middle of a compliation sequence. The list is used to free all blocks in
case compilation fails.
yjit_gen_block() is renamed to gen_single_block() to make it distinct from
gen_block_version(). Call to limit_block_version() and block_t
allocation is moved into the function to help tidy error checking in the
outer loop.
limit_block_version() now returns by value. I feel that an out parameter
with conditional mutation is unnecessarily hard to read in code that
does not need to go for last drop performance. There is a good chance
that the optimizer is able to output identical code anyways.
Previously, YJIT assumed that it's always possible to generate a new
basic block when servicing a stub in branch_stub_hit(). When YJIT is out
of executable memory, for example, this assumption doesn't hold up.
Add handling to branch_stub_hit() for servicing stubs without consuming
more executable memory by adding a code path that exits to the
interpreter at the location the branch stub represents. The new code
path reconstructs interpreter state in branch_stub_hit() and then exits
with a new snippet called `code_for_exit_from_stub` that returns
`Qundef` from the YJIT native stack frame.
As this change adds another place where we regenerate code from
`branch_t`, extract the logic for it into a new function and call it
regenerate_branch(). While we are at it, make the branch shrinking code
path in branch_stub_hit() more explicit.
This new functionality is hard to test without full support for out of
memory conditions. To verify this change, I ran
`RUBY_YJIT_ENABLE=1 make check -j12` with the following patch to stress
test the new code path:
```diff
diff --git a/yjit_core.c b/yjit_core.c
index 4ab63d9806..5788b8c5ed 100644
--- a/yjit_core.c
+++ b/yjit_core.c
@@ -878,8 +878,12 @@ branch_stub_hit(branch_t *branch, const uint32_t target_idx, rb_execution_contex
cb_set_write_ptr(cb, branch->end_addr);
}
+if (rand() < RAND_MAX/2) {
// Compile the new block version
p_block = gen_block_version(target, target_ctx, ec);
+}else{
+ p_block = NULL;
+}
if (!p_block && branch_modified) {
// We couldn't generate a new block for the branch, but we modified the branch.
```
We can enable the new test along with other OOM tests once full support
lands.
Other small changes:
* yjit_utils.c (print_str): Update to work with new native frame shape.
Follow up for 8fa0ee4d40.
* yjit_iface.c (rb_yjit_init): Run yjit_init_core() after
yjit_init_codegen() so `cb` and `ocb` are available.
* YJIT: Implement optimized_method_struct_aref
* YJIT: Implement struct_aref without method call
Struct member reads can be compiled directly into a memory read (with
either one or two levels of indirection).
* YJIT: Implement optimized struct aset
* YJIT: Update tests for struct access
* YJIT: Add counters for remaining optimized methods
* Check for INT32_MAX overflow
It only takes a struct with 0x7fffffff/8+1 members. Also add some
cheap compile time checks.
* Add tests for non-embedded struct aref/aset
Co-authored-by: Alan Wu <XrXr@users.noreply.github.com>
This commit adds an entry_exit field to block_t for use in
invalidate_block_version(). By patching the start of the block while
invalidating it, invalidate_block_version() can function correctly
while there is no executable memory left for new branch stubs.
This change additionally fixes correctness for situations where we
cannot patch incoming jumps to the invalidated block. In situations
such as Shopify/yjit#226, the address to the start of the block
is saved and used later, possibly after the block is invalidated.
The assume_* family of function now generate block->entry_exit before
remembering blocks for invalidation.
RubyVM::YJIT.simulate_oom! is introduced for testing out of memory
conditions. The test for it is disabled for now because OOM triggers
other failure conditions not addressed by this commit.
FixesShopify/yjit#226
This is the minimal correct objtostring implementation in YJIT.
For correctness, it is important that to_string not get called on strings or subclasses of string.
There is a new test for this behavior.
A follow up should implement an optimized version for other types as performed in `vm_objtostring`.
Co-authored-by: John Hawthorn <jhawthorn@github.com>
Co-authored-by: John Hawthorn <jhawthorn@github.com>
* YJIT: Support iseq sends with mixed kwargs
Co-authored-by: Kevin Newton <kddnewton@gmail.com>
* Add additional comments to iseq sends
Co-authored-by: Kevin Newton <kddnewton@gmail.com>
* YJIT: Support kwargs sends with all defaults
Previously keyword argument methods were only compiled by YJIT when all
keywords were specified in the caller.
This adds support for calling methods with keyword arguments when no
keyword arguments are specified and all are filled with the defaults.
* Remove unused send_iseq_kwargs_none_passed
`duphash` showed up in the top-20 most frequent exit ops for @jhawthorn's benchmark that renders github.com/about
The implementation was almost exactly the same as `duparray`
Co-authored-by: John Hawthorn <john@hawthorn.email>
Co-authored-by: John Hawthorn <john@hawthorn.email>
844588f915 made it so that trying to call
gc_verify_compaction_references on unsupported platform result in an
exception rather than a crash. Rescue the exception in a YJIT btest
that uses gc_verify_compaction_references.
2d98593bf5 made it so that
attr_accessor methods fire C method tracing events.
Previously, we weren't checking for whether we are tracing before
compiling, leading to missed events.
Since global invalidation invalidates all code, and that attr_accessor
methods can never enable tracing while running, events are only dropped
when YJIT tries to compile when tracing is already enabled.
Factor out the code for checking tracing and check it before generating
code for attr_accessor methods.
This change fixes TestSetTraceFunc#test_tracepoint_attr when it's
ran in isolation.
Previously, we were shuffling keyword arguments before checking for
interrupts. In the case that we side exit in the interrupt check,
we left the interpreter with an already-shuffled argument list for
the call, resulting in a double shuffle, leaving the locals in the
wrong order for the callee.
Do keyword shuffling after all the possible side exits.
Co-authored-by: Kevin Newton <kddnewton@gmail.com>
In an effort to simplify the logic YJIT generates for accessing instance
variable, YJIT ensures that a given name-to-index mapping exists at
compile time. In the case that the mapping doesn't exist, it was created
by using rb_ivar_set() with Qundef on the sample object we see at
compile time. This hack isn't fine if the sample object happens to be
frozen, in which case YJIT would raise a FrozenError unexpectedly.
To deal with this, make a new function that only reserves the mapping
but doesn't touch the object. This is rb_obj_ensure_iv_index_mapping().
This new function superceeds the functionality of rb_iv_index_tbl_lookup()
so it was removed.
Reported by and includes a test case from John Hawthorn <john@hawthorn.email>
Fixes: GH-282
When calling "to_s" on an instance of String, the method simply returns
self. In this situation most of the work comes from setting up the
method call. It turns out that both railsbench and liquid-render do this
a lot.
When generating code for opt_send_without_block, we already generate a
known class guard, so we can detect when the receiver is a String
instance. Since gen_send_cfunc() is also used for gen_invokesuper(), and
gen_invokesuper() doesn't generate a known class guard, a new nullable
parameter for specialized codegen function is added.
Closes GH-245
This change fixes some cases where YJIT fails to fire tracing events.
Most of the situations YJIT did not handle correctly involves enabling
tracing while running inside generated code.
A new operation to invalidate all generated code is added, which uses
patching to make generated code exit at the next VM instruction
boundary. A new routine called `jit_prepare_routine_call()` is
introduced to facilitate this and should be used when generating code
that could allocate, or could otherwise use `RB_VM_LOCK_ENTER()`.
The `c_return` event is fired in the middle of an instruction as opposed
to at an instruction boundary, so it requires special handling. C method
call return points are patched to go to a fucntion which does everything
the interpreter does, including firing the `c_return` event. The
generated code for C method calls normally does not fire the event.
Invalided code should not change after patching so the exits are not
clobbered. A new variable is introduced to track the region of code that
should not change.
Adds yjit support for setting global variables.
Co-authored-by: Aaron Patterson <tenderlove@ruby-lang.org>
Co-authored-by: John Hawthorn <john@hawthorn.email>
Methods with optional parameters don't always start executing at the
first PC, but we compile all methods assuming that they do. This commit
adds a guard to ensure that we're actually starting at the first PC for
methods with optional params
As an optimization, multiple objects could share the same singleton
class. The optimization introduced in 6698e43393
wasn't handling this correctly so was generating guards that never pass
for the inputs we defer compilation to wait for. After generating
identical code multiple times and failing, the call site is falsely
recognized as megamorphic and it side exits. See disassembly for the
following before this commit:
def foo(obj)
obj.itself
end
o = Object.new.singleton_class
foo(o)
puts YJIT.disasm(method(:foo))
See also: comment in rb_singleton_class_clone_and_attach().
Singleton classes should only ever be attached to one object. This means
that checking for the object should be the same as checking for the
class. This should be slightly faster by avoiding one memory acccess as
well as allowing us to skip checking if the receiver is a heap object.
This will be most common for calling class methods.
This commit improves opt_not by making it correct when TrueClass#!
and/or FalseClass#! is defined and genearting better code when the
receiver is a heap object.
guard_known_class() can now handle true, false, and nil, and we
introduce a codegen function reimplementing rb_obj_not(), used when we
know we are calling into rb_obj_not().
Co-authored-by: Maxime Chevalier-Boisvert <maximechevalierb@gmail.com>
Co-authored-by: Noah Gibbs <the.codefolio.guy@gmail.com>
invokebuiltin_delegate is a special version of invokebuiltin used for
sending a contiguous subset of the current method's locals.
In some cases YJIT would already handle this for trivial cases it could
be inlined, implementing this OP allows it to work when the method isn't
inlinable (not marked as 'inline', does more than just call, not called
from yjit, etc).
This commit implements the topn instruction
Co-Authored-By: Maxime Chevalier-Boisvert <maxime.chevalierboisvert@shopify.com>
Co-Authored-By: Noah Gibbs <noah.gibbs@shopify.com>
* Implement gen_newarray
* Implement newhash for n=0
* Add yjit tests for newhash/newarray
* Fix integer size warning on clang
* Save PC and SP in newhash and newarray
Co-authored-by: Maxime Chevalier-Boisvert <maximechevalierb@gmail.com>
* Implement getblockparamproxy
* Parallel runner: wait for timeout thread to terminate after killing
Or else the leak cheaker could sees the thread as running and cause test
failures in test-tool.
* Add a comment, use jne
* Comment about where 0x3 comes from
Takashi Kokubun
Maple Ong
Jimmy Miller
Alan Wu
Matthew Draper
John Hawthorn
Maple Ong
Zack Deveau
Matt Valentine-House
Noah Gibbs (and/or Benchmark CI)
Alan Wu
Jeremy Evans
Adam Hess
Ian C. Anderson
Koichi Sasada
Kevin Newton
Aaron Patterson
Maxime Chevalier-Boisvert
eileencodes
Kevin Deisz
Maxime Chevalier-Boisvert