Tested on production workloads at Shopify for > 1 year and proven
to be quite stable. Enabling YJIT at run-time is still guarded
behind the --yjit command-line option for now.
Raise a `SyntaxError` with the parser error message, in the case
reading from a file instead of the `-e` option or standard input. So
syntax_suggest can get the message from the caught error.
This solves multiple problems.
First, RB_VM_LOCK_ENTER/LEAVE is a barrier. We could at least use the
_NO_BARRIER variant.
Second, this doesn't need to interfere with GC or other GVL users when
multiple Ractors are used. This needs to be used in very few places, so
the benefit of fine-grained locking would outweigh its small maintenance
cost.
Third, it fixes a crash for YJIT. Because YJIT is never disabled until a
process exits unlike MJIT that finishes earlier, we could call jit_cont_free
when EC no longer exists, which crashes RB_VM_LOCK_ENTER.
Adds the `syntax_suggest` syntax error display tool to Ruby through the same mechanism as `error_highlight` and `did_you_mean`. Reference ticket: https://bugs.ruby-lang.org/issues/18159close#4845
## What is syntax_suggest?
When a syntax error is raised by requiring a file, dead_end will use a combination of indentation and lexing to identify the problem.
> Note: Previously this tool was named `dead_end`.
## Known issues
- SyntaxSearch's approach of showing syntax errors only works through integration with `require`, `load`, `autoload`, and `require_relative` (since it monkeypatches them to detect syntax errors). It does not work with direct Ruby file invocations https://github.com/zombocom/dead_end/issues/31.
- This causes failure in the test suite (test_expected_backtrace_location_when_inheriting_from_basic_object_and_including_kernel) and confusion when inspecting backtraces if there's a different error when trying to require a file such as measuring memory (https://github.com/zombocom/syntax_suggest/issues/124#issuecomment-1006705016).
- Discussed fix. We previously talked about opening up `SyntaxError` to be monkeypatched in the same way that other gems hook into `NoMethodError`. This is currently not possible and requires development work. When we last talked about it at RubyKaigi Nobu expressed an ability to make such a change.
* Simplify around `USE_YJIT` macro
- Use `USE_YJIT` macro only instead of `YJIT_BUILD`.
- An intermediate macro `YJIT_SUPPORTED_P` is no longer used.
* Bail out if YJIT is enabled on unsupported platforms
rb_ary_tmp_new suggests that the array is temporary in some way, but
that's not true, it just creates an array that's hidden and not on the
transient heap. This commit renames it to rb_ary_hidden_new.
YJIT is now a build-time opt-in so on platforms that YJIT could support
it could still be unavailable due to user discretion. Use MJIT for --jit
and don't display YJIT related command line options in --help when YJIT
is not included in the build.
Since enabling YJIT or MJIT drastically changes what could go wrong at
runtime, it's good to be front and center about whether they are enabled
when dumping a crash report. Previously, `RUBY_DESCRIPTION` and the
description printed when crashing can be different when a JIT is on.
Introduce a new internal data global, `rb_dynamic_description`, and set
it to be the same as `RUBY_DESCRIPTION` during initialization; use it
when crashing.
* version.c: Init_ruby_description(): Initialize and use
`rb_dynamic_description`.
* error.c: Change crash reports to use `rb_dynamic_description`.
* ruby.c: Call `Init_ruby_description()` earlier. Slightly more work
for when we exit right after printing the description but that
was deemed acceptable.
* include/ruby/version.h: Talk about how JIT info is not in
`ruby_description`.
* test/-ext-/bug_reporter/test_bug_reporter.rb: Remove handling for
crash description being different from `RUBY_DESCRIPTION`.
* test/ruby/test_rubyoptions.rb: ditto
Co-authored-by: Nobuyoshi Nakada <nobu@ruby-lang.org>
Co-authored-by: Alan Wu <alanwu@ruby-lang.org>
In December 2021, we opened an [issue] to solicit feedback regarding the
porting of the YJIT codebase from C99 to Rust. There were some
reservations, but this project was given the go ahead by Ruby core
developers and Matz. Since then, we have successfully completed the port
of YJIT to Rust.
The new Rust version of YJIT has reached parity with the C version, in
that it passes all the CRuby tests, is able to run all of the YJIT
benchmarks, and performs similarly to the C version (because it works
the same way and largely generates the same machine code). We've even
incorporated some design improvements, such as a more fine-grained
constant invalidation mechanism which we expect will make a big
difference in Ruby on Rails applications.
Because we want to be careful, YJIT is guarded behind a configure
option:
```shell
./configure --enable-yjit # Build YJIT in release mode
./configure --enable-yjit=dev # Build YJIT in dev/debug mode
```
By default, YJIT does not get compiled and cargo/rustc is not required.
If YJIT is built in dev mode, then `cargo` is used to fetch development
dependencies, but when building in release, `cargo` is not required,
only `rustc`. At the moment YJIT requires Rust 1.60.0 or newer.
The YJIT command-line options remain mostly unchanged, and more details
about the build process are documented in `doc/yjit/yjit.md`.
The CI tests have been updated and do not take any more resources than
before.
The development history of the Rust port is available at the following
commit for interested parties:
1fd9573d8b
Our hope is that Rust YJIT will be compiled and included as a part of
system packages and compiled binaries of the Ruby 3.2 release. We do not
anticipate any major problems as Rust is well supported on every
platform which YJIT supports, but to make sure that this process works
smoothly, we would like to reach out to those who take care of building
systems packages before the 3.2 release is shipped and resolve any
issues that may come up.
[issue]: https://bugs.ruby-lang.org/issues/18481
Co-authored-by: Maxime Chevalier-Boisvert <maximechevalierb@gmail.com>
Co-authored-by: Noah Gibbs <the.codefolio.guy@gmail.com>
Co-authored-by: Kevin Newton <kddnewton@gmail.com>
Add a hook point to initialize extra extension libraries. The default
hook function is replaced when linking a strong `Init_extra_exts`
symbol. A builder can insert an object file that defines Init_extra_exts
by XLDFLAGS.
```
compiling ../ruby.c
../ruby.c:1547:17: error: implicit declaration of function 'setup_yjit_options' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
setup_yjit_options(s, &opt->yjit);
^
../ruby.c:1547:17: note: did you mean 'setup_mjit_options'?
../ruby.c:1122:1: note: 'setup_mjit_options' declared here
setup_mjit_options(const char *s, struct mjit_options *mjit_opt)
^
../ruby.c:1547:45: error: no member named 'yjit' in 'struct ruby_cmdline_options'; did you mean 'mjit'?
setup_yjit_options(s, &opt->yjit);
^~~~
mjit
../ruby.c:192:25: note: 'mjit' declared here
struct mjit_options mjit;
^
../ruby.c:1924:28: error: no member named 'yjit' in 'struct ruby_cmdline_options'; did you mean 'mjit'?
rb_yjit_init(&opt->yjit);
^~~~
mjit
../ruby.c:192:25: note: 'mjit' declared here
struct mjit_options mjit;
^
3 errors generated.
```
* Rename --jit to --mjit
[Feature #18349]
* Fix a few more --jit references
* Fix MJIT Actions
* More s/jit/mjit/ and re-introduce --disable-jit
* Update NEWS.md
* Fix test_bug_reporter_add
* Add --yjit-no-type-prop so we can test YJIT without type propagation
* Fix typo in command line option
* Leave just two test workflows enable for YJIT
Takashi Kokubun
Maxime Chevalier-Boisvert
Nobuyoshi Nakada
yui-knk
schneems
Yuta Saito
Peter Zhu
Alan Wu
Chris Seaton
Kazuhiro NISHIYAMA
Yusuke Endoh
Aaron Patterson