This commit changes RUBY_GC_HEAP_INIT_SIZE_{40,80,160,320,640}_SLOTS to
RUBY_GC_HEAP_{0,1,2,3,4}_INIT_SLOTS. This is easier to use because the
user does not need to determine the slot sizes (which can vary between
32 and 64 bit systems). They now just use the heap names
(`GC.stat_heap.keys`).
In general, if the same option specifying a single value is given
multiple times at the same level, the last one overrides the earlier
ones, unless prohibited.
There’s no reason to prevent RUBYOPT from controlling the backtrace
limit. In fact, Matz said [0] he was expecting this to be possible.
[0] https://bugs.ruby-lang.org/issues/8661#note-27
Introduce Universal Parser mode for the parser.
This commit includes these changes:
* Introduce `UNIVERSAL_PARSER` macro. All of CRuby related functions
are passed via `struct rb_parser_config_struct` when this macro is enabled.
* Add CI task with 'cppflags=-DUNIVERSAL_PARSER' for ubuntu.
Instead of `rb_fork`, to update `current_fork_gen` which has been
introduced at 3563e1383f.
Otherwise, the forked process attempts to stop the timer thread, but
raises an exception because the thread is not alive in the child and
dies because already no tag is present at that time.
Followup: ac123f167a
RB_WARN_CATEGORY_ALL_BITS is exposed in a public header, so it
makes sense for it to be updated to contain all valid bits.
Instead we introduce RB_WARN_CATEGORY_DEFAULT_BITS to list the
categories that are enabled by default.
[Feature #19538]
This new `peformance` warning category is disabled by default.
It needs to be specifically enabled via `-W:performance` or `Warning[:performance] = true`
Add a sampling option to trace exits
Running YJIT with trace exits enabled can make very large metrics files.
This allows us to configure a sample rate to make tracing exits possible
on larger tests. This also updates the documented YJIT options.
Co-authored-by: Alan Wu <XrXr@users.noreply.github.com>
Co-authored-by: John Hawthorn <john@hawthorn.email>
Co-authored-by: Maxime Chevalier-Boisvert <maximechevalierb@gmail.com>
Remove !USE_RVARGC code
[Feature #19579]
The Variable Width Allocation feature was turned on by default in Ruby
3.2. Since then, we haven't received bug reports or backports to the
non-Variable Width Allocation code paths, so we assume that nobody is
using it. We also don't plan on maintaining the non-Variable Width
Allocation code, so we are going to remove it.
This reverts commit 35136e1e9c.
test-spec has been failing since this revision.
.github/workflows/compilers.yml:82
https://github.com/ruby/ruby/actions/runs/4276884159/jobs/7445299562
```
env:
# Minimal flags to pass the check.
default_cc: 'gcc-11 -fcf-protection -Wa,--generate-missing-build-notes=yes'
optflags: '-O2'
LDFLAGS: '-Wl,-z,now'
# FIXME: Drop skipping options
# https://bugs.ruby-lang.org/issues/18061
# https://sourceware.org/annobin/annobin.html/Test-pie.html
TEST_ANNOCHECK_OPTS: "--skip-pie --skip-gaps"
```
Failure:
```
1)
An exception occurred during: Kernel#require (file extensions) does not load a C-extension file if a complex-extensioned .rb file is already loaded
/__w/ruby/ruby/src/spec/ruby/core/kernel/shared/require.rb:317
Kernel#require (file extensions) does not load a C-extension file if a complex-extensioned .rb file is already loaded ERROR
LeakError: Leaked file descriptor: 8 : #<File:/__w/ruby/ruby/src/spec/ruby/fixtures/code/load_fixture.ext.rb>
/__w/ruby/ruby/src/spec/ruby/core/kernel/require_spec.rb:5:in `<top (required)>'
2)
An exception occurred during: Kernel#require ($LOADED_FEATURES) stores an absolute path
/__w/ruby/ruby/src/spec/ruby/core/kernel/shared/require.rb:330
Kernel#require ($LOADED_FEATURES) stores an absolute path ERROR
LeakError: Closed file descriptor: 8
/__w/ruby/ruby/src/spec/ruby/core/kernel/require_spec.rb:5:in `<top (required)>'
3)
An exception occurred during: Kernel#require ($LOADED_FEATURES) does not load a non-canonical path for a file already loaded
/__w/ruby/ruby/src/spec/ruby/core/kernel/shared/require.rb:535
Kernel#require ($LOADED_FEATURES) does not load a non-canonical path for a file already loaded ERROR
LeakError: Leaked file descriptor: 8 : #<File:/__w/ruby/ruby/src/spec/ruby/fixtures/code/load_fixture.rb>
/__w/ruby/ruby/src/spec/ruby/core/kernel/require_spec.rb:5:in `<top (required)>'
4)
An exception occurred during: Kernel#require ($LOADED_FEATURES) does not load a ../ relative path for a file already loaded
/__w/ruby/ruby/src/spec/ruby/core/kernel/shared/require.rb:551
Kernel#require ($LOADED_FEATURES) does not load a ../ relative path for a file already loaded ERROR
LeakError: Leaked file descriptor: 9 : #<File:../code/load_fixture.rb>
/__w/ruby/ruby/src/spec/ruby/core/kernel/require_spec.rb:5:in `<top (required)>'
5)
An exception occurred during: Kernel#require ($LOADED_FEATURES) complex, enumerator, rational, thread, ruby2_keywords are already required
/__w/ruby/ruby/src/spec/ruby/core/kernel/shared/require.rb:563
Kernel#require ($LOADED_FEATURES) complex, enumerator, rational, thread, ruby2_keywords are already required ERROR
LeakError: Closed file descriptor: 8
Closed file descriptor: 9
/__w/ruby/ruby/src/spec/ruby/core/kernel/require_spec.rb:5:in `<top (required)>'
6)
An exception occurred during: Kernel.require (file extensions) does not load a C-extension file if a complex-extensioned .rb file is already loaded
/__w/ruby/ruby/src/spec/ruby/core/kernel/shared/require.rb:317
Kernel.require (file extensions) does not load a C-extension file if a complex-extensioned .rb file is already loaded ERROR
LeakError: Leaked file descriptor: 8 : #<File:/__w/ruby/ruby/src/spec/ruby/fixtures/code/load_fixture.ext.rb>
/__w/ruby/ruby/src/spec/ruby/core/kernel/require_spec.rb:23:in `<top (required)>'
7)
An exception occurred during: Kernel.require ($LOADED_FEATURES) stores an absolute path
/__w/ruby/ruby/src/spec/ruby/core/kernel/shared/require.rb:330
Kernel.require ($LOADED_FEATURES) stores an absolute path ERROR
LeakError: Closed file descriptor: 8
/__w/ruby/ruby/src/spec/ruby/core/kernel/require_spec.rb:23:in `<top (required)>'
8)
An exception occurred during: Kernel.require ($LOADED_FEATURES) does not load a non-canonical path for a file already loaded
/__w/ruby/ruby/src/spec/ruby/core/kernel/shared/require.rb:535
Kernel.require ($LOADED_FEATURES) does not load a non-canonical path for a file already loaded ERROR
LeakError: Leaked file descriptor: 8 : #<File:/__w/ruby/ruby/src/spec/ruby/fixtures/code/load_fixture.rb>
/__w/ruby/ruby/src/spec/ruby/core/kernel/require_spec.rb:23:in `<top (required)>'
9)
An exception occurred during: Kernel.require ($LOADED_FEATURES) does not load a ../ relative path for a file already loaded
/__w/ruby/ruby/src/spec/ruby/core/kernel/shared/require.rb:551
Kernel.require ($LOADED_FEATURES) does not load a ../ relative path for a file already loaded ERROR
LeakError: Leaked file descriptor: 9 : #<File:../code/load_fixture.rb>
/__w/ruby/ruby/src/spec/ruby/core/kernel/require_spec.rb:23:in `<top (required)>'
10)
An exception occurred during: Kernel.require ($LOADED_FEATURES) complex, enumerator, rational, thread, ruby2_keywords are already required
/__w/ruby/ruby/src/spec/ruby/core/kernel/shared/require.rb:563
Kernel.require ($LOADED_FEATURES) complex, enumerator, rational, thread, ruby2_keywords are already required ERROR
LeakError: Closed file descriptor: 8
Closed file descriptor: 9
/__w/ruby/ruby/src/spec/ruby/core/kernel/require_spec.rb:23:in `<top (required)>'
11)
An exception occurred during: Kernel#require_relative with a relative path (file extensions) does not load a C-extension file if a complex-extensioned .rb file is already loaded
/__w/ruby/ruby/src/spec/ruby/core/kernel/require_relative_spec.rb:197
Kernel#require_relative with a relative path (file extensions) does not load a C-extension file if a complex-extensioned .rb file is already loaded ERROR
LeakError: Leaked file descriptor: 8 : #<File:/__w/ruby/ruby/src/spec/ruby/fixtures/code/load_fixture.ext.rb>
/__w/ruby/ruby/src/spec/ruby/core/kernel/require_relative_spec.rb:4:in `<top (required)>'
12)
An exception occurred during: Kernel#require_relative with a relative path ($LOADED_FEATURES) stores an absolute path
/__w/ruby/ruby/src/spec/ruby/core/kernel/require_relative_spec.rb:205
Kernel#require_relative with a relative path ($LOADED_FEATURES) stores an absolute path ERROR
LeakError: Closed file descriptor: 8
/__w/ruby/ruby/src/spec/ruby/core/kernel/require_relative_spec.rb:4:in `<top (required)>'
13)
An exception occurred during: Kernel#require_relative with an absolute path (file extensions) does not load a C-extension file if a complex-extensioned .rb file is already loaded
/__w/ruby/ruby/src/spec/ruby/core/kernel/require_relative_spec.rb:399
Kernel#require_relative with an absolute path (file extensions) does not load a C-extension file if a complex-extensioned .rb file is already loaded ERROR
LeakError: Leaked file descriptor: 8 : #<File:/__w/ruby/ruby/src/spec/ruby/fixtures/code/load_fixture.ext.rb>
/__w/ruby/ruby/src/spec/ruby/core/kernel/require_relative_spec.rb:277:in `<top (required)>'
14)
An exception occurred during: Kernel#require_relative with an absolute path ($LOAD_FEATURES) stores an absolute path
/__w/ruby/ruby/src/spec/ruby/core/kernel/require_relative_spec.rb:407
Kernel#require_relative with an absolute path ($LOAD_FEATURES) stores an absolute path ERROR
LeakError: Closed file descriptor: 8
/__w/ruby/ruby/src/spec/ruby/core/kernel/require_relative_spec.rb:277:in `<top (required)>'
```
When loading Ruby source files, we can save the result of
successful opens as open(2)/openat(2) are a fairly expensive
syscalls. This also avoids a time-of-check-to-time-of-use
(TOCTTOU) problem.
This reduces open(2) syscalls during `require'; but should be
most apparent when users have a small $LOAD_PATH. Users with
large $LOAD_PATH will benefit less since there'll be more
open(2) failures due to ENOENT.
With `strace -c -e openat ruby -e exit' under Linux, this
results in a ~14% reduction of openat(2) syscalls
(glibc uses openat(2) to implement open(2)).
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
0.00 0.000000 0 296 110 openat
0.00 0.000000 0 254 110 openat
Additionally, the introduction of `struct ruby_file_load_state'
may make future optimizations more apparent.
This change cannot benefit binary (.so) loading since the
dlopen(3) API requires a filename and I'm not aware of an
alternative that takes a pre-existing FD. In typical
situations, Ruby source files outnumber the mount of .so
files.
Tested on production workloads at Shopify for > 1 year and proven
to be quite stable. Enabling YJIT at run-time is still guarded
behind the --yjit command-line option for now.
Raise a `SyntaxError` with the parser error message, in the case
reading from a file instead of the `-e` option or standard input. So
syntax_suggest can get the message from the caught error.
This solves multiple problems.
First, RB_VM_LOCK_ENTER/LEAVE is a barrier. We could at least use the
_NO_BARRIER variant.
Second, this doesn't need to interfere with GC or other GVL users when
multiple Ractors are used. This needs to be used in very few places, so
the benefit of fine-grained locking would outweigh its small maintenance
cost.
Third, it fixes a crash for YJIT. Because YJIT is never disabled until a
process exits unlike MJIT that finishes earlier, we could call jit_cont_free
when EC no longer exists, which crashes RB_VM_LOCK_ENTER.
Adds the `syntax_suggest` syntax error display tool to Ruby through the same mechanism as `error_highlight` and `did_you_mean`. Reference ticket: https://bugs.ruby-lang.org/issues/18159close#4845
## What is syntax_suggest?
When a syntax error is raised by requiring a file, dead_end will use a combination of indentation and lexing to identify the problem.
> Note: Previously this tool was named `dead_end`.
## Known issues
- SyntaxSearch's approach of showing syntax errors only works through integration with `require`, `load`, `autoload`, and `require_relative` (since it monkeypatches them to detect syntax errors). It does not work with direct Ruby file invocations https://github.com/zombocom/dead_end/issues/31.
- This causes failure in the test suite (test_expected_backtrace_location_when_inheriting_from_basic_object_and_including_kernel) and confusion when inspecting backtraces if there's a different error when trying to require a file such as measuring memory (https://github.com/zombocom/syntax_suggest/issues/124#issuecomment-1006705016).
- Discussed fix. We previously talked about opening up `SyntaxError` to be monkeypatched in the same way that other gems hook into `NoMethodError`. This is currently not possible and requires development work. When we last talked about it at RubyKaigi Nobu expressed an ability to make such a change.
* Simplify around `USE_YJIT` macro
- Use `USE_YJIT` macro only instead of `YJIT_BUILD`.
- An intermediate macro `YJIT_SUPPORTED_P` is no longer used.
* Bail out if YJIT is enabled on unsupported platforms
rb_ary_tmp_new suggests that the array is temporary in some way, but
that's not true, it just creates an array that's hidden and not on the
transient heap. This commit renames it to rb_ary_hidden_new.
YJIT is now a build-time opt-in so on platforms that YJIT could support
it could still be unavailable due to user discretion. Use MJIT for --jit
and don't display YJIT related command line options in --help when YJIT
is not included in the build.
Since enabling YJIT or MJIT drastically changes what could go wrong at
runtime, it's good to be front and center about whether they are enabled
when dumping a crash report. Previously, `RUBY_DESCRIPTION` and the
description printed when crashing can be different when a JIT is on.
Introduce a new internal data global, `rb_dynamic_description`, and set
it to be the same as `RUBY_DESCRIPTION` during initialization; use it
when crashing.
* version.c: Init_ruby_description(): Initialize and use
`rb_dynamic_description`.
* error.c: Change crash reports to use `rb_dynamic_description`.
* ruby.c: Call `Init_ruby_description()` earlier. Slightly more work
for when we exit right after printing the description but that
was deemed acceptable.
* include/ruby/version.h: Talk about how JIT info is not in
`ruby_description`.
* test/-ext-/bug_reporter/test_bug_reporter.rb: Remove handling for
crash description being different from `RUBY_DESCRIPTION`.
* test/ruby/test_rubyoptions.rb: ditto
Co-authored-by: Nobuyoshi Nakada <nobu@ruby-lang.org>
Co-authored-by: Alan Wu <alanwu@ruby-lang.org>
In December 2021, we opened an [issue] to solicit feedback regarding the
porting of the YJIT codebase from C99 to Rust. There were some
reservations, but this project was given the go ahead by Ruby core
developers and Matz. Since then, we have successfully completed the port
of YJIT to Rust.
The new Rust version of YJIT has reached parity with the C version, in
that it passes all the CRuby tests, is able to run all of the YJIT
benchmarks, and performs similarly to the C version (because it works
the same way and largely generates the same machine code). We've even
incorporated some design improvements, such as a more fine-grained
constant invalidation mechanism which we expect will make a big
difference in Ruby on Rails applications.
Because we want to be careful, YJIT is guarded behind a configure
option:
```shell
./configure --enable-yjit # Build YJIT in release mode
./configure --enable-yjit=dev # Build YJIT in dev/debug mode
```
By default, YJIT does not get compiled and cargo/rustc is not required.
If YJIT is built in dev mode, then `cargo` is used to fetch development
dependencies, but when building in release, `cargo` is not required,
only `rustc`. At the moment YJIT requires Rust 1.60.0 or newer.
The YJIT command-line options remain mostly unchanged, and more details
about the build process are documented in `doc/yjit/yjit.md`.
The CI tests have been updated and do not take any more resources than
before.
The development history of the Rust port is available at the following
commit for interested parties:
1fd9573d8b
Our hope is that Rust YJIT will be compiled and included as a part of
system packages and compiled binaries of the Ruby 3.2 release. We do not
anticipate any major problems as Rust is well supported on every
platform which YJIT supports, but to make sure that this process works
smoothly, we would like to reach out to those who take care of building
systems packages before the 3.2 release is shipped and resolve any
issues that may come up.
[issue]: https://bugs.ruby-lang.org/issues/18481
Co-authored-by: Maxime Chevalier-Boisvert <maximechevalierb@gmail.com>
Co-authored-by: Noah Gibbs <the.codefolio.guy@gmail.com>
Co-authored-by: Kevin Newton <kddnewton@gmail.com>
Add a hook point to initialize extra extension libraries. The default
hook function is replaced when linking a strong `Init_extra_exts`
symbol. A builder can insert an object file that defines Init_extra_exts
by XLDFLAGS.
```
compiling ../ruby.c
../ruby.c:1547:17: error: implicit declaration of function 'setup_yjit_options' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
setup_yjit_options(s, &opt->yjit);
^
../ruby.c:1547:17: note: did you mean 'setup_mjit_options'?
../ruby.c:1122:1: note: 'setup_mjit_options' declared here
setup_mjit_options(const char *s, struct mjit_options *mjit_opt)
^
../ruby.c:1547:45: error: no member named 'yjit' in 'struct ruby_cmdline_options'; did you mean 'mjit'?
setup_yjit_options(s, &opt->yjit);
^~~~
mjit
../ruby.c:192:25: note: 'mjit' declared here
struct mjit_options mjit;
^
../ruby.c:1924:28: error: no member named 'yjit' in 'struct ruby_cmdline_options'; did you mean 'mjit'?
rb_yjit_init(&opt->yjit);
^~~~
mjit
../ruby.c:192:25: note: 'mjit' declared here
struct mjit_options mjit;
^
3 errors generated.
```
* Rename --jit to --mjit
[Feature #18349]
* Fix a few more --jit references
* Fix MJIT Actions
* More s/jit/mjit/ and re-introduce --disable-jit
* Update NEWS.md
* Fix test_bug_reporter_add
* Add --yjit-no-type-prop so we can test YJIT without type propagation
* Fix typo in command line option
* Leave just two test workflows enable for YJIT
TestRubyOptions#test_enable was broken on OpenBSD after the yjit
merge. --yjit (and --enable-all, which enables --yjit) fails on
OpenBSD because yjit uses an insecure mmap call (both writable
and executable), in alloc_exec_mem, which OpenBSD does not allow.
This can probably be reverted if yjit switches to a more secure
mmap design (writable xor executable). This would involve
initially calling mmap with PROT_READ | PROT_WRITE, and after writing
of executable code has finished, using mprotect to switch to
PROT_READ | PROT_EXEC. I believe Firefox uses this approach for
their Javascript engine since Firefox 46.
Previously, options such as "--yjit123" would enable YJIT. Additionally,
the error message for argument parsing mentioned "--jit-..." instead of
"--yjit-...".
Must not be a bad idea to improve documents. [ci skip]
In fact many functions declared in the header file are already
documented more or less. They were just copy & pasted, with applying
some style updates.
When runtime_libruby_path does not include '/', it attempts to call
rb_str_resize with negative length. This change makes sure that the
length non-negative.
Co-Authored-By: xtkoba (Tee KOBAYASHI) <xtkoba+ruby@gmail.com>
Also document that both :deprecated and :experimental are supported
:category option values.
The locations where warnings were marked as deprecation warnings
was previously reviewed by shyouhei.
Comment a couple locations where deprecation warnings should probably
be used but are not currently used because deprecation warning
enablement has not occurred at the time they are called
(RUBY_FREE_MIN, RUBY_HEAP_MIN_SLOTS, -K).
Add assert_deprecated_warn to test assertions. Use this to simplify
some tests, and fix failing tests after marking some warnings with
deprecated category.
This changes the behavior, which I'm not sure is acceptable.
However, it's odd to allow an option to be combined, but change
the behavior of the option when combined.
* Use UTF-8 as default for Encoding.default_external on Windows
* Document UTF-8 change on Windows to Encoding.default_external
fix https://bugs.ruby-lang.org/issues/16604
Peter Zhu
Nobuyoshi Nakada
Jeremy Evans
Benoit Daloze
Tom Stuart
Takashi Kokubun
yui-knk
Yla Aioi
Jean Boussier
Adam Hess
Eric Wong
Maxime Chevalier-Boisvert
schneems
Yuta Saito
Alan Wu
Chris Seaton
Kazuhiro NISHIYAMA
Yusuke Endoh
Aaron Patterson
Alan Wu
Noah Gibbs
Maxime Chevalier-Boisvert
Jose Narvaez
卜部昌平
S-H-GAMELINKS
Samuel Williams
NARUSE, Yui
YO4 (Yoshinao Muramatsu)
Kenta Murata
Lars Kanis
Koichi Sasada