This dispatches to a c func for doing the dynamic lookup. I experimented with chain on the proc but wasn't able to detect which call sites would be monomorphic vs polymorphic. There is definitely room for optimization here, but it does reduce exits.
This allows x86_64 based YJIT to run on Docker Desktop on Apple silicon (arm64)
Mac because it will avoid a subtle behavior difference in `mprotect` system call
between the Linux kernel and `qemu-x86_64` user space emulator.
* Implement optimize send in yjit
This successfully makes all our benchmarks exit way less for optimize send reasons.
It makes some benchmarks faster, but not by as much as I'd like. I think this implementation
works, but there are definitely more optimial arrangements. For example, what if we compiled
send to a jump table? That seems like perhaps the most optimal we could do, but not obvious (to me)
how to implement give our current setup.
Co-authored-by: Alan Wu <XrXr@users.noreply.github.com>
* Attempt at fixing the issues raised by @XrXr
* fix allowlist
* returns 0 instead of nil when not found
* remove comment about encoding exception
* Fix up c changes
* Update assert
Co-authored-by: Alan Wu <XrXr@users.noreply.github.com>
* get rid of unneeded code and fix the flags
* Apply suggestions from code review
Co-authored-by: Alan Wu <XrXr@users.noreply.github.com>
* rename and fix typo
Co-authored-by: Alan Wu <XrXr@users.noreply.github.com>
* YJIT: fix a parameter name
* YJIT: add support for calling bmethods
This commit adds support for the VM_METHOD_TYPE_BMETHOD method type in
YJIT. You can get these type of methods from facilities like
Kernel#define_singleton_method and Module#define_method.
Even though the body of these methods are blocks, the parameter setup
for them is exactly the same as VM_METHOD_TYPE_ISEQ, so we can reuse
the same logic in gen_send_iseq(). You can see this from how
vm_call_bmethod() eventually calls setup_parameters_complex() with
arg_setup_method.
Bmethods do need their frame environment to be setup differently. We
handle this by allowing callers of gen_send_iseq() to control the iseq,
the frame flag, and the prev_ep. The `prev_ep` goes into the same
location as the block handler would go into in an iseq method frame.
Co-authored-by: John Hawthorn <john@hawthorn.email>
Co-authored-by: John Hawthorn <john@hawthorn.email>
* Initial support for VM_CALL_ARGS_SPLAT
This implements support for calls with splat (*) for some methods. In
benchmarks this made very little difference for most benchmarks, but a
large difference for binarytrees. Looking at side exits, many
benchmarks now don't exit for splat, but exit for some other
reason. Binarytrees however had a number of calls that used splat args
that are now much faster. In my non-scientific benchmarking this made
splat args performance on par with not using splat args at all.
* Fix wording and whitespace
Co-authored-by: Maxime Chevalier-Boisvert <maximechevalierb@gmail.com>
* Get rid of side_effect reassignment
Co-authored-by: Maxime Chevalier-Boisvert <maximechevalierb@gmail.com>
Previously we cleared the cache for all the code in the system when we
flip memory protection, which was prohibitively expensive since the
operation is not constant time. Instead, only clear the cache for the
memory region of newly written code when we write out new code.
This brings the runtime for the 30k_if_else test down to about 6 seconds
from the previous 45 seconds on my laptop.
We have a large extern block in cruby.rs leftover from the port. We can
use bindgen for it now and reserve the manual declaration for just a
handful of vm_insnhelper.c functions.
Fixup a few minor discrepencies bindgen found between the C declaration
and the manual declaration. Mostly missing `const` on the C side.
Teach getblockparamproxy to handle the no-block case without exiting
Co-authored-by: John Hawthorn <john@hawthorn.email>
Co-authored-by: John Hawthorn <john@hawthorn.email>
In a small script the speed of this feature isn't really noticeable but
on Rails it's very noticeable how slow this can be. This PR aims to
speed up two parts of the functionality.
1) The Rust exit recording code
Instead of adding all samples as we see them to the yjit_raw_samples and
yjit_line_samples, we can increment the counter on the ones we've seen
before. This will be faster on traces where we are hitting the same
stack often. In a crude measurement of booting just the active record
base test (`test/cases/base_test.rb`) we found that this improved the
speed by 1 second.
This also results in a smaller marshal dump file which sped up the test
boot time by 4 seconds with trace exits on.
2) The Ruby parsing code
Previously we were allocating new arrays using `shift` and
`each_with_index`. This change avoids allocating new arrays by using an
index. This change saves us the most amount of time, gaining 11 seconds.
Before this change the test boot time took 62 seconds, after it took 47
seconds. This is still too long but it's a step closer to faster
functionality. Next we're going to tackle allowing you to collect trace
exits for a specific instruction. There is also some potential slowness
in the GC code that I'd like to take a second look at.
Co-authored-by: Aaron Patterson <tenderlove@ruby-lang.org>
Co-authored-by: Aaron Patterson <tenderlove@ruby-lang.org>
This commit makes YJIT allocate memory for generated code gradually as
needed. Previously, YJIT allocates all the memory it needs on boot in
one go, leading to higher than necessary resident set size (RSS) and
time spent on boot initializing the memory with a large memset().
Users should no longer need to search for a magic number to pass to
`--yjit-exec-mem` since physical memory consumption should now more
accurately reflect the requirement of the workload.
YJIT now reserves a range of addresses on boot. This region start out
with no access permission at all so buggy attempts to jump to the region
crashes like before this change. To get this hardening at finer
granularity than the page size, we fill each page with trapping
instructions when we first allocate physical memory for the page.
Most of the time applications don't need 256 MiB of executable code, so
allocating on-demand ends up doing less total work than before. Case in
point, a simple `ruby --yjit-call-threshold=1 -eitself` takes about
half as long after this change. In terms of memory consumption, here is
a table to give a rough summary of the impact:
| Peak RSS in MiB | -eitself example | railsbench once |
| :-------------: | ---------------: | --------------: |
| before | 265 | 377 |
| after | 11 | 143 |
| no YJIT | 10 | 101 |
A new module is introduced to handle allocation bookkeeping.
`CodePtr` is moved into the module since it has a close relationship
with the new `VirtualMemory` struct. This new interface has a slightly
smaller surface than before in that marking a region as writable is no
longer a public operation.
When running with `--yjit-stats` turned on, yjit can inform the user
what the most common exits are. While this is useful information it
doesn't tell you the source location of the code that exited or what the
code that exited looks like. This change intends to fix that.
To use the feature, run yjit with the `--yjit-trace-exits` option,
which will record the backtrace for every exit that occurs. This functionality
requires the stats feature to be turned on. Calling `--yjit-trace-exits`
will automatically set the `--yjit-stats` option.
Users must call `RubyVM::YJIT.dump_exit_locations(filename)` which will
Marshal dump the contents of `RubyVM::YJIT.exit_locations` into a file
based on the passed filename.
*Example usage:*
Given the following script, we write to a file called
`concat_array.dump` the results of `RubyVM::YJIT.exit_locations`.
```ruby
def concat_array
["t", "r", *x = "u", "e"].join
end
1000.times do
concat_array
end
RubyVM::YJIT.dump_exit_locations("concat_array.dump")
```
When we run the file with this branch and the appropriate flags the
stacktrace will be recorded. Note Stackprof needs to be installed or you
need to point to the library directly.
```
./ruby --yjit --yjit-call-threshold=1 --yjit-trace-exits -I/Users/eileencodes/open_source/stackprof/lib test.rb
```
We can then read the dump file with Stackprof:
```
./ruby -I/Users/eileencodes/open_source/stackprof/lib/ /Users/eileencodes/open_source/stackprof/bin/stackprof --text concat_array.dump
```
Results will look similar to the following:
```
==================================
Mode: ()
Samples: 1817 (0.00% miss rate)
GC: 0 (0.00%)
==================================
TOTAL (pct) SAMPLES (pct) FRAME
1001 (55.1%) 1001 (55.1%) concatarray
335 (18.4%) 335 (18.4%) invokeblock
178 (9.8%) 178 (9.8%) send
140 (7.7%) 140 (7.7%) opt_getinlinecache
...etc...
```
Simply inspecting the `concatarray` method will give `SOURCE
UNAVAILABLE` because the source is insns.def.
```
./ruby -I/Users/eileencodes/open_source/stackprof/lib/ /Users/eileencodes/open_source/stackprof/bin/stackprof --text concat_array.dump --method concatarray
```
Result:
```
concatarray (nonexistent.def:1)
samples: 1001 self (55.1%) / 1001 total (55.1%)
callers:
1000 ( 99.9%) Object#concat_array
1 ( 0.1%) Gem.suffixes
callees (0 total):
code:
SOURCE UNAVAILABLE
```
However if we go deeper to the callee we can see the exact
source of the `concatarray` exit.
```
./ruby -I/Users/eileencodes/open_source/stackprof/lib/ /Users/eileencodes/open_source/stackprof/bin/stackprof --text concat_array.dump --method Object#concat_array
```
```
Object#concat_array (/Users/eileencodes/open_source/rust_ruby/test.rb:1)
samples: 0 self (0.0%) / 1000 total (55.0%)
callers:
1000 ( 100.0%) block in <main>
callees (1000 total):
1000 ( 100.0%) concatarray
code:
| 1 | def concat_array
1000 (55.0%) | 2 | ["t", "r", *x = "u", "e"].join
| 3 | end
```
The `--walk` option is recommended for this feature as it make it
easier to traverse the tree of exits.
*Goals of this feature:*
This feature is meant to give more information when working on YJIT.
The idea is that if we know what code is exiting we can decide what
areas to prioritize when fixing exits. In some cases this means adding
prioritizing avoiding certain exits in yjit. In more complex cases it
might mean changing the Ruby code to be more performant when run with
yjit. Ultimately the more information we have about what code is exiting
AND why, the better we can make yjit.
*Known limitations:*
* Due to tracing exits, running this on large codebases like Rails
can be quite slow.
* On complex methods it can still be difficult to pinpoint the exact cause of
an exit.
* Stackprof is a requirement to to view the backtrace information from
the dump file.
Co-authored-by: Aaron Patterson <tenderlove@ruby-lang.org>
Co-authored-by: Aaron Patterson <tenderlove@ruby-lang.org>
For string concat, see if compile-time encoding of strings matches.
If so, use simple buffer string concat at runtime. Otherwise, use
encoding-checking string concat.
In December 2021, we opened an [issue] to solicit feedback regarding the
porting of the YJIT codebase from C99 to Rust. There were some
reservations, but this project was given the go ahead by Ruby core
developers and Matz. Since then, we have successfully completed the port
of YJIT to Rust.
The new Rust version of YJIT has reached parity with the C version, in
that it passes all the CRuby tests, is able to run all of the YJIT
benchmarks, and performs similarly to the C version (because it works
the same way and largely generates the same machine code). We've even
incorporated some design improvements, such as a more fine-grained
constant invalidation mechanism which we expect will make a big
difference in Ruby on Rails applications.
Because we want to be careful, YJIT is guarded behind a configure
option:
```shell
./configure --enable-yjit # Build YJIT in release mode
./configure --enable-yjit=dev # Build YJIT in dev/debug mode
```
By default, YJIT does not get compiled and cargo/rustc is not required.
If YJIT is built in dev mode, then `cargo` is used to fetch development
dependencies, but when building in release, `cargo` is not required,
only `rustc`. At the moment YJIT requires Rust 1.60.0 or newer.
The YJIT command-line options remain mostly unchanged, and more details
about the build process are documented in `doc/yjit/yjit.md`.
The CI tests have been updated and do not take any more resources than
before.
The development history of the Rust port is available at the following
commit for interested parties:
1fd9573d8b
Our hope is that Rust YJIT will be compiled and included as a part of
system packages and compiled binaries of the Ruby 3.2 release. We do not
anticipate any major problems as Rust is well supported on every
platform which YJIT supports, but to make sure that this process works
smoothly, we would like to reach out to those who take care of building
systems packages before the 3.2 release is shipped and resolve any
issues that may come up.
[issue]: https://bugs.ruby-lang.org/issues/18481
Co-authored-by: Maxime Chevalier-Boisvert <maximechevalierb@gmail.com>
Co-authored-by: Noah Gibbs <the.codefolio.guy@gmail.com>
Co-authored-by: Kevin Newton <kddnewton@gmail.com>
* Rename --jit to --mjit
[Feature #18349]
* Fix a few more --jit references
* Fix MJIT Actions
* More s/jit/mjit/ and re-introduce --disable-jit
* Update NEWS.md
* Fix test_bug_reporter_add
As part of YJIT's strategy for promoting Ruby constant expressions into
constants in the output native code, the interpreter calls
rb_yjit_constant_ic_update() from opt_setinlinecache.
The block invalidation loop indirectly calls rb_darray_remove_unordered(),
which does a shuffle remove. Because of this, looping with an
incrementing counter like done previously can miss some elements in the
array. Repeatedly invalidate the first element instead.
The bug this commit resolves does not seem to cause crashes or divergent
behaviors.
Co-authored-by: Jemma Issroff <jemmaissroff@gmail.com>
On Rails we're seeing a lot of exits for ivars in the Active Record
tests. In trying to track them down it was hard to find what code is
exiting.
This change adds a counted exit for when an object is "megamorphic". In
these cases there are too many specializations in the Ruby code so YJIT
exits.
Co-authored-by: Aaron Patterson tenderlove@ruby-lang.org
Previously, YJIT crashes with rb_bug() when asked to compile new methods
while out of executable memory.
To handle this situation gracefully, this change keeps track of all the
blocks compiled each invocation in case YJIT runs out of memory in the
middle of a compliation sequence. The list is used to free all blocks in
case compilation fails.
yjit_gen_block() is renamed to gen_single_block() to make it distinct from
gen_block_version(). Call to limit_block_version() and block_t
allocation is moved into the function to help tidy error checking in the
outer loop.
limit_block_version() now returns by value. I feel that an out parameter
with conditional mutation is unnecessarily hard to read in code that
does not need to go for last drop performance. There is a good chance
that the optimizer is able to output identical code anyways.
Previously, YJIT assumed that it's always possible to generate a new
basic block when servicing a stub in branch_stub_hit(). When YJIT is out
of executable memory, for example, this assumption doesn't hold up.
Add handling to branch_stub_hit() for servicing stubs without consuming
more executable memory by adding a code path that exits to the
interpreter at the location the branch stub represents. The new code
path reconstructs interpreter state in branch_stub_hit() and then exits
with a new snippet called `code_for_exit_from_stub` that returns
`Qundef` from the YJIT native stack frame.
As this change adds another place where we regenerate code from
`branch_t`, extract the logic for it into a new function and call it
regenerate_branch(). While we are at it, make the branch shrinking code
path in branch_stub_hit() more explicit.
This new functionality is hard to test without full support for out of
memory conditions. To verify this change, I ran
`RUBY_YJIT_ENABLE=1 make check -j12` with the following patch to stress
test the new code path:
```diff
diff --git a/yjit_core.c b/yjit_core.c
index 4ab63d9806..5788b8c5ed 100644
--- a/yjit_core.c
+++ b/yjit_core.c
@@ -878,8 +878,12 @@ branch_stub_hit(branch_t *branch, const uint32_t target_idx, rb_execution_contex
cb_set_write_ptr(cb, branch->end_addr);
}
+if (rand() < RAND_MAX/2) {
// Compile the new block version
p_block = gen_block_version(target, target_ctx, ec);
+}else{
+ p_block = NULL;
+}
if (!p_block && branch_modified) {
// We couldn't generate a new block for the branch, but we modified the branch.
```
We can enable the new test along with other OOM tests once full support
lands.
Other small changes:
* yjit_utils.c (print_str): Update to work with new native frame shape.
Follow up for 8fa0ee4d40.
* yjit_iface.c (rb_yjit_init): Run yjit_init_core() after
yjit_init_codegen() so `cb` and `ocb` are available.
* YJIT: Implement optimized_method_struct_aref
* YJIT: Implement struct_aref without method call
Struct member reads can be compiled directly into a memory read (with
either one or two levels of indirection).
* YJIT: Implement optimized struct aset
* YJIT: Update tests for struct access
* YJIT: Add counters for remaining optimized methods
* Check for INT32_MAX overflow
It only takes a struct with 0x7fffffff/8+1 members. Also add some
cheap compile time checks.
* Add tests for non-embedded struct aref/aset
Co-authored-by: Alan Wu <XrXr@users.noreply.github.com>
* YJIT: Support kwargs sends with all defaults
Previously keyword argument methods were only compiled by YJIT when all
keywords were specified in the caller.
This adds support for calling methods with keyword arguments when no
keyword arguments are specified and all are filled with the defaults.
* Remove unused send_iseq_kwargs_none_passed
In an effort to minimize build issues on non x64 platforms, we can
decide at build time to not build the bulk of YJIT. This should fix
obscure build errors like this one on riscv64:
yjit_asm.c:137:(.text+0x3fa): relocation truncated to fit: R_RISCV_PCREL_HI20 against `alloc_exec_mem'
We also don't need to bulid YJIT on `--disable-jit-support` builds.
One wrinkle to this is that the YJIT Ruby module will not be defined
when YJIT is stripped from the build. I think that's a fair change as
it's only meant to be used for YJIT development.
For upstreaming, we want functions we export either prefixed with "rb_"
or made static. Historically we haven't been following this rule, so we
were "leaking" a lot of symbols as `make leak-globals` would tell us.
This change unifies everything YJIT into a single compilation unit,
yjit.o, and makes everything unprefixed static to pass `make leak-globals`.
This manual "unified build" setup is similar to that of vm.o.
Having everything in one compilation unit allows static functions to
be visible across YJIT files and removes the need for declarations in
headers in some cases. Unnecessary declarations were removed.
Other changes of note:
- switched to MJIT_SYMBOL_EXPORT_BEGIN which indicates stuff as being
off limits for native extensions
- the first include of each YJIT file is change to be "internal.h"
- undefined MAP_STACK before explicitly redefining it since it
collide's with a definition in system headers. Consider renaming?
Takashi Kokubun
Jimmy Miller
Noah Gibbs
Matthew Draper
Tatsuya Kawano
Alan Wu
John Hawthorn
Maxime Chevalier-Boisvert
Noah Gibbs
Nobuyoshi Nakada
Eileen M. Uchitelle
Dave Schwantes
Alan Wu
Takayoshi Nishida
eileencodes
Kevin Newton