darray was used in YJIT which required the functions to not trigger GC.
YJIT has now moved to Rust and does not use darray anymore, so we can
remove the functions that don't trigger GC and only keep the ones that
trigger GC.
In December 2021, we opened an [issue] to solicit feedback regarding the
porting of the YJIT codebase from C99 to Rust. There were some
reservations, but this project was given the go ahead by Ruby core
developers and Matz. Since then, we have successfully completed the port
of YJIT to Rust.
The new Rust version of YJIT has reached parity with the C version, in
that it passes all the CRuby tests, is able to run all of the YJIT
benchmarks, and performs similarly to the C version (because it works
the same way and largely generates the same machine code). We've even
incorporated some design improvements, such as a more fine-grained
constant invalidation mechanism which we expect will make a big
difference in Ruby on Rails applications.
Because we want to be careful, YJIT is guarded behind a configure
option:
```shell
./configure --enable-yjit # Build YJIT in release mode
./configure --enable-yjit=dev # Build YJIT in dev/debug mode
```
By default, YJIT does not get compiled and cargo/rustc is not required.
If YJIT is built in dev mode, then `cargo` is used to fetch development
dependencies, but when building in release, `cargo` is not required,
only `rustc`. At the moment YJIT requires Rust 1.60.0 or newer.
The YJIT command-line options remain mostly unchanged, and more details
about the build process are documented in `doc/yjit/yjit.md`.
The CI tests have been updated and do not take any more resources than
before.
The development history of the Rust port is available at the following
commit for interested parties:
1fd9573d8b
Our hope is that Rust YJIT will be compiled and included as a part of
system packages and compiled binaries of the Ruby 3.2 release. We do not
anticipate any major problems as Rust is well supported on every
platform which YJIT supports, but to make sure that this process works
smoothly, we would like to reach out to those who take care of building
systems packages before the 3.2 release is shipped and resolve any
issues that may come up.
[issue]: https://bugs.ruby-lang.org/issues/18481
Co-authored-by: Maxime Chevalier-Boisvert <maximechevalierb@gmail.com>
Co-authored-by: Noah Gibbs <the.codefolio.guy@gmail.com>
Co-authored-by: Kevin Newton <kddnewton@gmail.com>
[Bug #18599]
`vm->loaded_features` and `vm->loaded_features_snapshot` both share the
same root. When a feature is pushed into `loaded_features`, the sharing
is broken and `loaded_features` is copied.
So an application requiring 1000 files, will allocate 1000 arrays of
increasing size, which is very wasteful.
To avoid this, we first clear the snapshot, so that `loaded_features`
can directly be pushed into.
Co-Authored-By: Peter Zhu <peter.zhu@shopify.com>
Using a fake (malloc) RArray is not friendly for the garbage
collector. Fake RArray does not have a heap page, so it causes Variable
Width Allocation to crash when we try to implement it on Arrays.
This commit changes feature_index from a RArray to a darray.
Previously this was being incorrectly swapped with TAG_RAISE in the next
line. This would end up checking the T_IMEMO throw_data to the exception
handling (which calls Module#===). This happened to not break existing
tests because Module#=== returned false when klass is NULL.
This commit handles throw from require correctly by jumping to the tag
retaining the TAG_THROW state.
Instead of always using a new anonymous module for Kernel#load if
the wrap argument is not false/nil, use the given module if a module
is provided.
Implements [Feature #6210]
Solaris CI still has a problem even with these commits, so it doesn't
appear to fix the issue. Reverting both 84e8e2a39bba874433b661bd378165bd03c9d6aa
and bfd2f159f0c60ef8ac5bce6042edd25a571769b7.
This appears to be only necessary on Solaris, but this commit
enables it unconditionally to test breakage. The following
commit will switch to only enabling it on Solaris.
This fixes issues with paths being loaded twice in certain cases
when symlinks are used.
It took me multiple attempts to get this working. My original
attempt tried to convert paths to realpaths before adding them
to $LOADED_FEATURES. Unfortunately, this doesn't work well
with the loaded feature index, which is based off load paths
and not realpaths. While I was able to get require working, I'm
fairly sure the loaded feature index was not being used as
expected, which would have significant performance implications.
Additionally, I was never able to get that approach working with
autoload when autoloading a non-realpath file. It also broke
some specs.
This takes a more conservative approach. Directly before loading the
file, if the file with the same realpath has been required, the
loading of the file is skipped. The realpaths are stored as
fstrings in a hidden hash.
When rebuilding the loaded feature index, the hash of realpaths
is also rebuilt. I'm guessing this makes rebuilding process
slower, but I don think that is a hot path. In general, modifying
loaded features is only done when reloading, and that tends to be
in non-production environments.
Change test_require_with_loaded_features_pop test to use 30 threads
and 300 iterations, instead of 4 threads and 1000 iterations.
I saw only sporadic failures with 4/1000, but consistent failures
30/300 threads. These failures were due to the fact that the
concurrent deletions from $LOADED_FEATURES in other threads can
result in rb_ary_entry returning nil when rebuilding the loaded
features index.
To avoid concurrency issues when rebuilding the loaded features
index, the building of the index itself is left alone, and
afterwards, a separate loop is done on a copy of the loaded feature
snapshot in order to rebuild the realpaths hash.
Fixes [Bug #17885]
Instead of relying on setting an unsetting ruby_verbose, which is
not thread-safe, restructure require_internal and load_lock to
accept a warn argument for whether to warn, and add
rb_require_internal_silent to require without warnings. Use
rb_require_internal_silent when loading encoding.
Note this does not modify ruby_debug and errinfo handling, those
remain thread-unsafe.
Also silent requires when loading transcoders.
This fixes issues with paths being loaded twice in certain cases
when symlinks are used.
It took me multiple attempts to get this working. My original
attempt tried to convert paths to realpaths before adding them
to $LOADED_FEATURES. Unfortunately, this doesn't work well
with the loaded feature index, which is based off load paths
and not realpaths. While I was able to get require working, I'm
fairly sure the loaded feature index was not being used as
expected, which would have significant performance implications.
Additionally, I was never able to get that approach working with
autoload when autoloading a non-realpath file. It also broke
some specs.
This takes a more conservative approach. Directly before loading the
file, if the file with the same realpath has been required, the
loading of the file is skipped. The realpaths are stored as
fstrings in a hidden hash.
When rebuilding the loaded feature index, the hash of realpaths
is also rebuilt. I'm guessing this makes rebuilding process
slower, but I don think that is a hot path. In general, modifying
loaded features is only done when reloading, and that tends to be
in non-production environments.
Change test_require_with_loaded_features_pop test to use 30 threads
and 300 iterations, instead of 4 threads and 1000 iterations.
I saw only sporadic failures with 4/1000, but consistent failures
30/300 threads. These failures were due to the fact that the
concurrent deletions from $LOADED_FEATURES in other threads can
result in rb_ary_entry returning nil when rebuilding the loaded
features index.
To avoid concurrency issues when rebuilding the loaded features
index, the building of the index itself is left alone, and
afterwards, a separate loop is done on a copy of the loaded feature
snapshot in order to rebuild the realpaths hash.
Fixes [Bug #17885]
If $LOADED_FEATURES is changed in the just required file, also the
index table needs to be updated before loaded_features_snapshot is
reset. If the snapshot is reset without updating the table, the
name of the added feature will not be found.
When attempting to require a file without an extension that has
already been required or provided with an .so extension, only
look for files with an .rb extension. There is no point in
trying to find files with an .so extension, since we already
know one has been loaded.
Previously, attempting to require such a file scanned the load
path twice, once for .rb and once for .so. Now it only scans
once for .rb. The scan once for .rb cannot be avoided, since
the .rb file would take precedence and should be loaded if it
exists.
Fixes [Bug #10902]
When looking for libraries to load with a feature name without
extension, `.rb` files are given priority. However, since the
feature index arrays were not in that order of priority, but in
the order in which they were loaded, a lower priority extension
library might be returned. In that case, the `.rb` file had to be
searched for again from the `$LOAD_PATH`, resulting in poor
performance.
C extensions can violate the ractor-safety, so only ractor-safe
C extensions (C methods) can run on non-main ractors.
rb_ext_ractor_safe(true) declares that the successive
defined methods are ractor-safe. Otherwiwze, defined methods
checked they are invoked in main ractor and raise an error
if invoked at non-main ractors.
[Feature #17307]
Update and format the Kernel#load documentation to separate the
three cases (absolute path, explicit relative path, other), and
also document that it raises LoadError on failure.
Fixes [Bug #16988]
Saves comitters' daily life by avoid #include-ing everything from
internal.h to make each file do so instead. This would significantly
speed up incremental builds.
We take the following inclusion order in this changeset:
1. "ruby/config.h", where _GNU_SOURCE is defined (must be the very
first thing among everything).
2. RUBY_EXTCONF_H if any.
3. Standard C headers, sorted alphabetically.
4. Other system headers, maybe guarded by #ifdef
5. Everything else, sorted alphabetically.
Exceptions are those win32-related headers, which tend not be self-
containing (headers have inclusion order dependencies).
TAG_FATAL represents interpreter closing state and ec->errinfo
contains FIXNUM (eTerminateSignal, etc). If we need to change the
state, then errinfo is also changed because TAG_RAISE assumes that
ec->errinfo contains a Exception object.
Without this patch, TAG_FATAL is ignored and no ec->errinfo change
so that it causes critical issue.
[Bug #16177]
This removes the security features added by $SAFE = 1, and warns for access
or modification of $SAFE from Ruby-level, as well as warning when calling
all public C functions related to $SAFE.
This modifies some internal functions that took a safe level argument
to no longer take the argument.
rb_require_safe now warns, rb_require_string has been added as a
version that takes a VALUE and does not warn.
One public C function that still takes a safe level argument and that
this doesn't warn for is rb_eval_cmd. We may want to consider
adding an alternative method that does not take a safe level argument,
and warn for rb_eval_cmd.
Looking at the list of symbols inside of libruby-static.a, I found
hundreds of functions that are defined, but used from nowhere.
There can be reasons for each of them (e.g. some functions are
specific to some platform, some are useful when debugging, etc).
However it seems the functions deleted here exist for no reason.
This changeset reduces the size of ruby binary from 26,671,456
bytes to 26,592,864 bytes on my machine.
We can check the function pointer passed to rb_define_global_function
like we do so in rb_define_method. It turns out that almost anybody
is misunderstanding the API.
After 5e86b005c0, I now think ANYARGS is
dangerous and should be extinct. There is only one usage of
MEMO::u3::func in load.c (where void Init_Foobar(vodi) is registered)
so why not just be explicit.
Benoit Daloze
Nobuyoshi Nakada
Peter Zhu
Alan Wu
Jean Boussier
John Hawthorn
Jeremy Evans
S-H-GAMELINKS
Samuel Williams
David Rodríguez
Koichi Sasada
Stefan Stüben
卜部昌平
MSP-Greg