This reduces the code size of libyjit.a by a lot. On darwin it went from
23 MiB to 12 MiB for me. I chose ThinLTO over fat LTO for the relatively
fast build time; in case we need to debug release-build-only problems
it won't be painful.
Ignoring the following messages:
```
(snip)
-e 'load "spec/bundler/support/bundle.rb"' -- install --gemfile=tool/bundler/dev_gems.rb
Using rake 13.0.6
Using bundler 2.5.0.dev
Using diff-lcs 1.5.0
Using parallel 1.22.1
Using parallel_tests 2.32.0
Using power_assert 2.0.2
Using rb_sys 0.9.52
Using rspec-support 3.12.0
Using rspec-core 3.12.0
Using rspec-expectations 3.12.0
Using rspec-mocks 3.12.1
Using test-unit 3.5.5
Using uri 0.12.0
Using webrick 1.7.0
Bundle complete! 11 Gemfile dependencies, 14 gems now installed.
Gems in the groups 'lint' and 'doc' were not installed.
Use `bundle info [gemname]` to see where a bundled gem is installed.
```
With --enable-yjit, you see an annoying warning like this:
warning: Missing auto-load script at offset 0 in section .debug_gdb_scripts
of file /home/k0kubun/src/github.com/ruby/ruby/.ruby/miniruby.
Use `info auto-load python-scripts [REGEXP]' to list them.
Using `rust-gdb` instead fixes it. I use this like `make gdb GDB=rust-gdb`.
If baseruby is available (and its version is different from one being
built) when compiling ruby, tool/outdate-bundled-gems.rb (which is
invoked by `make install`) wrongly deletes debug.so and rbs_extension.so
in .bundle/extension/*.
This leads to a broken installation of ruby which lacks the libraries,
which may make rubygems show the following warnings (in some additional
complex conditions):
```
$ irb
Ignoring debug-1.7.1 because its extensions are not built. Try: gem pristine debug --version 1.7.1
Ignoring rbs-2.8.2 because its extensions are not built. Try: gem pristine rbs --version 2.8.2
irb(main):001:0>
```
According to some committers, tool/outdate-bundled-gems.rb is introduced
for fixing a build issue, but the detail is not recorded. The issue
seems to occur only when debug gem or rbs gem is updated, so it is
difficult to fix the script so soon.
Tentatively, this change stops invoking the script by default.
This should be backported to ruby_3_2.
Fixes [Bug #19271]
I noticed this while running test_yjit with --mjit-call-threshold=1,
which redefines `Integer#<`. When Ruby is monkey-patched,
MJIT itself could be broken.
Similarly, Ruby scripts could break MJIT in many different ways. I
prepared the same set of hooks as YJIT so that we could possibly
override it and disable it on those moments. Every constant under
RubyVM::MJIT is private and thus it's an unsupported behavior though.
SIZE_POOL_COUNT is a GC macro, it should belong in gc.h and not shape.h.
SIZE_POOL_COUNT doesn't depend on shape.h so we can have shape.h depend
on gc.h.
Co-Authored-By: Matt Valentine-House <matt@eightbitraptor.com>
* YJIT: Generate debug info in release builds
They are helpful in case we need to do core dump debugging.
* Remove Cirrus DOC skip rule
The syntax for this is weird, and escaping [ and ] cause parse failures.
Cirrus' docs said to surround with .*, but then that seems to skip
everything. Revert e0a4205eb7 for now.
I see several arguments in doing so.
First they use a non trivial amount of memory, so for various memory
profiling/mapping tools it is relevant to have visibility of the space
occupied by shapes.
Then, some pathological code can create a tons of shape, so it is
valuable to have a way to have a way to observe shapes without having
to compile Ruby with `SHAPE_DEBUG=1`.
And additionally it's likely much faster to dump then this way than
to use `RubyVM::Shape`.
There are however a few open questions:
- Shapes can't respect the `since:` argument. Not sure what to do when
it is provided. Would probably make sense to not dump them.
- Maybe it would make more sense to have a separate `ObjectSpace.dump_shapes`?
- Maybe instead `dump_all` should take a `shapes: false` argument?
Additionally, `ObjectSpace.dump_shapes` is added for the use case of
debugging the evolution of the shape tree.
Cases like this:
```ruby
obj = Object.new
loop do
obj.instance_variable_set(:@foo, 1)
obj.remove_instance_variable(:@foo)
end
```
can cause us to use many more shapes than we want (and even run out).
This commit changes the code such that when an instance variable is
removed, we'll walk up the shape tree, find the shape, then rebuild any
child nodes that happened to be below the "targetted for removal" IV.
This also requires moving any instance variables so that indexes derived
from the shape tree will work correctly.
Co-Authored-By: Jemma Issroff <jemmaissroff@gmail.com>
Co-authored-by: John Hawthorn <jhawthorn@github.com>
This commit adds a `capacity` field to shapes, and adds shape
transitions whenever an object's capacity changes. Objects which are
allocated out of a bigger size pool will also make a transition from the
root shape to the shape with the correct capacity for their size pool
when they are allocated.
This commit will allow us to remove numiv from objects completely, and
will also mean we can guarantee that if two objects share shapes, their
IVs are in the same positions (an embedded and extended object cannot
share shapes). This will enable us to implement ivar sets in YJIT using
object shapes.
Co-Authored-By: Aaron Patterson <tenderlove@ruby-lang.org>
Object Shapes is used for accessing instance variables and representing the
"frozenness" of objects. Object instances have a "shape" and the shape
represents some attributes of the object (currently which instance variables are
set and the "frozenness"). Shapes form a tree data structure, and when a new
instance variable is set on an object, that object "transitions" to a new shape
in the shape tree. Each shape has an ID that is used for caching. The shape
structure is independent of class, so objects of different types can have the
same shape.
For example:
```ruby
class Foo
def initialize
# Starts with shape id 0
@a = 1 # transitions to shape id 1
@b = 1 # transitions to shape id 2
end
end
class Bar
def initialize
# Starts with shape id 0
@a = 1 # transitions to shape id 1
@b = 1 # transitions to shape id 2
end
end
foo = Foo.new # `foo` has shape id 2
bar = Bar.new # `bar` has shape id 2
```
Both `foo` and `bar` instances have the same shape because they both set
instance variables of the same name in the same order.
This technique can help to improve inline cache hits as well as generate more
efficient machine code in JIT compilers.
This commit also adds some methods for debugging shapes on objects. See
`RubyVM::Shape` for more details.
For more context on Object Shapes, see [Feature: #18776]
Co-Authored-By: Aaron Patterson <tenderlove@ruby-lang.org>
Co-Authored-By: Eileen M. Uchitelle <eileencodes@gmail.com>
Co-Authored-By: John Hawthorn <john@hawthorn.email>
Object Shapes is used for accessing instance variables and representing the
"frozenness" of objects. Object instances have a "shape" and the shape
represents some attributes of the object (currently which instance variables are
set and the "frozenness"). Shapes form a tree data structure, and when a new
instance variable is set on an object, that object "transitions" to a new shape
in the shape tree. Each shape has an ID that is used for caching. The shape
structure is independent of class, so objects of different types can have the
same shape.
For example:
```ruby
class Foo
def initialize
# Starts with shape id 0
@a = 1 # transitions to shape id 1
@b = 1 # transitions to shape id 2
end
end
class Bar
def initialize
# Starts with shape id 0
@a = 1 # transitions to shape id 1
@b = 1 # transitions to shape id 2
end
end
foo = Foo.new # `foo` has shape id 2
bar = Bar.new # `bar` has shape id 2
```
Both `foo` and `bar` instances have the same shape because they both set
instance variables of the same name in the same order.
This technique can help to improve inline cache hits as well as generate more
efficient machine code in JIT compilers.
This commit also adds some methods for debugging shapes on objects. See
`RubyVM::Shape` for more details.
For more context on Object Shapes, see [Feature: #18776]
Co-Authored-By: Aaron Patterson <tenderlove@ruby-lang.org>
Co-Authored-By: Eileen M. Uchitelle <eileencodes@gmail.com>
Co-Authored-By: John Hawthorn <john@hawthorn.email>
This add support for bmake, which should allow building with
`configure --enable-yjit` for the BSDs. Tested on FreeBSD 13 and
on macOS with `configure MAKE=bmake` on a case-sensitive file system.
It works by including a fragment into the Makefile through the configure
script, similar to common.mk. It uses the always rebuild approach to
keep build system changes minimal.
Non-GNU make seems to generate empty revision.h, but it doesn't make
sense since https://github.com/ruby/ruby/pull/6382.
Also the $(HAVE_BASERUBY:yes=tmp) hack doesn't seem to be working on
OpenBSD. I'll remove it to focus on fixing RubyCI first, and then deal
with baseruby-missing environments. At least a snapshot should have
revision.h and it might work fine though.
Fixes id.h error during updating ripper.c by `make after-update`.
While it used to update id.h in the build directory, but was trying to
update ripper.c in the source directory. In principle, files in the
source directory can or should not depend on files in the build
directory.
Previously YARV bytecode implemented constant caching by having a pair
of instructions, opt_getinlinecache and opt_setinlinecache, wrapping a
series of getconstant calls (with putobject providing supporting
arguments).
This commit replaces that pattern with a new instruction,
opt_getconstant_path, handling both getting/setting the inline cache and
fetching the constant on a cache miss.
This is implemented by storing the full constant path as a
null-terminated array of IDs inside of the IC structure. idNULL is used
to signal an absolute constant reference.
$ ./miniruby --dump=insns -e '::Foo::Bar::Baz'
== disasm: #<ISeq:<main>@-e:1 (1,0)-(1,13)> (catch: FALSE)
0000 opt_getconstant_path <ic:0 ::Foo::Bar::Baz> ( 1)[Li]
0002 leave
The motivation for this is that we had increasingly found the need to
disassemble the instructions between the opt_getinlinecache and
opt_setinlinecache in order to determine the constant we are fetching,
or otherwise store metadata.
This disassembly was done:
* In opt_setinlinecache, to register the IC against the constant names
it is using for granular invalidation.
* In rb_iseq_free, to unregister the IC from the invalidation table.
* In YJIT to find the position of a opt_getinlinecache instruction to
invalidate it when the cache is populated
* In YJIT to register the constant names being used for invalidation.
With this change we no longe need disassemly for these (in fact
rb_iseq_each is now unused), as the list of constant names being
referenced is held in the IC. This should also make it possible to make
more optimizations in the future.
This may also reduce the size of iseqs, as previously each segment
required 32 bytes (on 64-bit platforms) for each constant segment. This
implementation only stores one ID per-segment.
There should be no significant performance change between this and the
previous implementation. Previously opt_getinlinecache was a "leaf"
instruction, but it included a jump (almost always to a separate cache
line). Now opt_getconstant_path is a non-leaf (it may
raise/autoload/call const_missing) but it does not jump. These seem to
even out.
I'm planning to introduce mjit_compiler.rb, and I want to make this
consistent with it. Consistency with compile.c doesn't seem important
for MJIT anyway.
* Optimize Marshal dump of large fixnum
Marshal's FIXNUM type only supports 31-bit fixnums, so on 64-bit
platforms the 63-bit fixnums need to be represented in Marshal's
BIGNUM.
Previously this was done by converting to a bugnum and serializing the
bignum object.
This commit avoids allocating the intermediate bignum object, instead
outputting the T_FIXNUM directly to a Marshal bignum. This maintains the
same representation as the previous implementation, including not using
LINKs for these large fixnums (an artifact of the previous
implementation always allocating a new BIGNUM).
This commit also avoids unnecessary st_lookups on immediate values,
which we know will not be in that table.
* Fastpath for loading FIXNUM from Marshal bignum
* Run update-deps
Creates simple bin stubs to load the extracted executable files.
After only extracted under `gems` directory, the gems are considered
installed but the executable scripts are not found.
Also the second argument is now the parent of the previous second and
third arguments.
Since #6006, we no longer avoid executing GC on mjit_worker.c and thus
there's no need to carefully change how we write code whether you're in
mjit.c or mjit_worker.c anymore.
The annocheck supports ELF format binaries compiled for any OS and for any
architecture. It can work in not only Linux but also FreeBSD and Solaris.
It is designed to be independent of the host OS and the architecture.
Even in Mac, as the binaries are Mach-O foramt, the annocheck fails correctly
with the message.
e.g. Test binaries compiled for Mac OSX 10.13.6 (target_os: darwin17) in Fedora 35.
```
$ cat /etc/fedora-release
Fedora release 35 (Thirty Five)
$ file ruby
ruby: Mach-O 64-bit x86_64 executable, flags:<NOUNDEFS|DYLDLINK|TWOLEVEL|WEAK_DEFINES|BINDS_TO_WEAK|PIE>
$ annocheck ruby
annocheck: Version 10.66.
annocheck: Warning: ruby: is not an ELF format file.
```
See <https://sourceware.org/bugzilla/show_bug.cgi?id=29173> for details.
In December 2021, we opened an [issue] to solicit feedback regarding the
porting of the YJIT codebase from C99 to Rust. There were some
reservations, but this project was given the go ahead by Ruby core
developers and Matz. Since then, we have successfully completed the port
of YJIT to Rust.
The new Rust version of YJIT has reached parity with the C version, in
that it passes all the CRuby tests, is able to run all of the YJIT
benchmarks, and performs similarly to the C version (because it works
the same way and largely generates the same machine code). We've even
incorporated some design improvements, such as a more fine-grained
constant invalidation mechanism which we expect will make a big
difference in Ruby on Rails applications.
Because we want to be careful, YJIT is guarded behind a configure
option:
```shell
./configure --enable-yjit # Build YJIT in release mode
./configure --enable-yjit=dev # Build YJIT in dev/debug mode
```
By default, YJIT does not get compiled and cargo/rustc is not required.
If YJIT is built in dev mode, then `cargo` is used to fetch development
dependencies, but when building in release, `cargo` is not required,
only `rustc`. At the moment YJIT requires Rust 1.60.0 or newer.
The YJIT command-line options remain mostly unchanged, and more details
about the build process are documented in `doc/yjit/yjit.md`.
The CI tests have been updated and do not take any more resources than
before.
The development history of the Rust port is available at the following
commit for interested parties:
1fd9573d8b
Our hope is that Rust YJIT will be compiled and included as a part of
system packages and compiled binaries of the Ruby 3.2 release. We do not
anticipate any major problems as Rust is well supported on every
platform which YJIT supports, but to make sure that this process works
smoothly, we would like to reach out to those who take care of building
systems packages before the 3.2 release is shipped and resolve any
issues that may come up.
[issue]: https://bugs.ruby-lang.org/issues/18481
Co-authored-by: Maxime Chevalier-Boisvert <maximechevalierb@gmail.com>
Co-authored-by: Noah Gibbs <the.codefolio.guy@gmail.com>
Co-authored-by: Kevin Newton <kddnewton@gmail.com>
Hiroshi SHIBATA
Takashi Kokubun
Alan Wu
Nobuyoshi Nakada
Yusuke Endoh
Benoit Daloze
Peter Zhu
Jean Boussier
Aaron Patterson
Daniel Colson
Martin Dürst
S-H-GAMELINKS
zverok
Jemma Issroff
Samuel Williams
John Hawthorn
Jean Boussier
Kevin Backhouse
Jun Aruga
Alan Wu