Граф коммитов

830 Коммитов

Автор SHA1 Сообщение Дата
Benoit Daloze 6abe20e87b Remove Encoding#replicate 2023-01-11 13:41:41 +01:00
Takashi Kokubun 9792d9e40f Revert "Re-enable test_ractor for YJIT"
This reverts commit 650a20a3e1.

Now that 3.2.0 is released, let's disable flaky tests. Koichi said he'll
rework Ractor implementation for this, and it has not been done yet.
2023-01-09 17:43:47 -08:00
Takashi Kokubun b9332ac8e7
MJIT: Cancel all on disastrous situations (#7019)
I noticed this while running test_yjit with --mjit-call-threshold=1, 
which redefines `Integer#<`. When Ruby is monkey-patched, 
MJIT itself could be broken.

Similarly, Ruby scripts could break MJIT in many different ways. I
prepared the same set of hooks as YJIT so that we could possibly
override it and disable it on those moments. Every constant under
RubyVM::MJIT is private and thus it's an unsupported behavior though.
2022-12-24 01:13:40 -08:00
Takashi Kokubun 67ef3cd3cc
Skip a flaky Ractor test for mswin 2022-12-21 16:56:41 -08:00
Takashi Kokubun 650a20a3e1
Re-enable test_ractor for YJIT
This would be still flaky, but we want to make sure there's no
YJIT-specific issue when Ruby 3.2 is released. We might skip it again
after the release.
2022-12-19 13:59:36 -08:00
Koichi Sasada ae19ac5b5b fixed encoding table
This reduces global lock acquiring for reading.
https://bugs.ruby-lang.org/issues/18949
2022-12-16 10:04:37 +09:00
Takashi Kokubun 307d2e0e91
Skip yet another flaky Ractor test 2022-12-13 22:40:13 -08:00
Takashi Kokubun 7b38853bc3
Skip another flaky Ractor test for YJIT 2022-12-02 10:26:54 -08:00
Takashi Kokubun 85f041c0c5
Skip a couple of Ractor tests
Koichi plans to rework Ractor implementation to address these failures.
He agreed to skip flaky Ractor tests for now.
2022-12-02 10:18:50 -08:00
Alan Wu eb2b717a8b
YJIT: Make case-when optimization respect === redefinition (#6846)
* YJIT: Make case-when optimization respect === redefinition

Even when a fixnum key is in the dispatch hash, if there is a case such
that its basic operations for === is redefined, we need to fall back to
checking each case like the interpreter. Semantically we're always
checking each case by calling === in order, it's just that this is not
observable when basic operations are intact.

When all the keys are fixnums, though, we can do the optimization we're
doing right now. Check for this condition.

* Update yjit/src/cruby_bindings.inc.rs

Co-authored-by: Takashi Kokubun <takashikkbn@gmail.com>

Co-authored-by: Takashi Kokubun <takashikkbn@gmail.com>
2022-12-02 11:40:16 -05:00
Alan Wu a0b0365e90 YJIT: Deallocate `struct Block` to plug memory leaks
Previously we essentially never freed block even after invalidation.
Their reference count never reached zero for a couple of reasons:
1. `Branch::block` formed a cycle with the block holding the branch
2. Strong count on a branch that has ever contained a stub never
   reached 0 because we increment the `.clone()` call for
   `BranchRef::into_raw()` didn't have a matching decrement.

It's not safe to immediately deallocate blocks during
invalidation since `branch_stub_hit()` can end up
running with a branch pointer from an invalidated branch.
To plug the leaks, we wait until code GC or global invalidation and
deallocate the blocks for iseqs that are definitely not running.
2022-11-30 12:23:50 -05:00
Jimmy Miller 98e9165b0a
Fix bug involving .send and overwritten methods. (#6752)
@casperisfine reporting a bug in this gist https://gist.github.com/casperisfine/d59e297fba38eb3905a3d7152b9e9350

After investigating I found it was caused by a combination of send and a c_func that we have overwritten in the JIT. For send calls, we need to do some stack manipulation before making the call. Because of the way exits works, we need to do that stack manipulation at the last possible moment. In this case, we weren't doing that stack manipulation at all. Unfortunately, with how the code is structured there isn't a great place to do that stack manipulation for our overridden C funcs.

Each overridden C func can return a boolean stating that it shouldn't be used. We would need to do the stack manipulation after all of those checks are done. We could pass a lambda(?) or separate out the logic for "can I run this override" from "now generate the code for it". Since we are coming up on a release, I went with the path of least resistence and just decided to not use these overrides if we are in a send call.

We definitely should revist this in the future.
2022-11-17 23:17:40 -05:00
Takashi Kokubun d15d1c01c2
Rename --mjit-min-calls to --mjit-call-threshold (#6731)
for consistency with YJIT
2022-11-14 23:38:52 -08:00
Alan Wu bc8ba244b8
YJIT: Fix invalidation for c_call and c_return (#6719)
Follow-up for 2b8191bdad. Since that
commit, we stopped doing code invalidation the second time the call and
return events are enabled. We need to do it every time these events are
enabled because we might have generated code while these events are
disabled.

Also rename locals and edit comments to make it more clear that the iseq
rewrite code path only happens the first time a particular iseq trace
event is enabled.
2022-11-13 12:51:19 -05:00
Nobuyoshi Nakada b02b8e7756
Let other test runners follow the change of GNU make 4.4 jobserver 2022-11-07 10:08:30 +09:00
Matthew Draper c746f380f2
YJIT: Support nil and blockparamproxy as blockarg in send (#6492)
Co-authored-by: John Hawthorn <john@hawthorn.email>

Co-authored-by: John Hawthorn <john@hawthorn.email>
2022-10-26 15:27:59 -04:00
Nobuyoshi Nakada 131c31a920
[Bug #19081] Show the caller location in warning for Ractor
The internal location in ractor.rb is not usefull at all.
```
$ ruby -e 'Ractor.new {}'
<internal:ractor>:267: warning: Ractor is experimental, ...
```
2022-10-26 19:43:14 +09:00
Jemma Issroff 9a5684bf7f Add test for ractor race condition on ivar sets 2022-10-14 11:59:36 -07:00
Jemma Issroff ad63b668e2
Revert "Revert "This commit implements the Object Shapes technique in CRuby.""
This reverts commit 9a6803c90b.
2022-10-11 08:40:56 -07:00
Alan Wu 7293bfe1bf
YJIT: add support for calling bmethods (#6489)
* YJIT: fix a parameter name

* YJIT: add support for calling bmethods

This commit adds support for the VM_METHOD_TYPE_BMETHOD method type in
YJIT. You can get these type of methods from facilities like
Kernel#define_singleton_method and Module#define_method.

Even though the body of these methods are blocks, the parameter setup
for them is exactly the same as VM_METHOD_TYPE_ISEQ, so we can reuse
the same logic in gen_send_iseq(). You can see this from how
vm_call_bmethod() eventually calls setup_parameters_complex() with
arg_setup_method.

Bmethods do need their frame environment to be setup differently. We
handle this by allowing callers of gen_send_iseq() to control the iseq,
the frame flag, and the prev_ep. The `prev_ep` goes into the same
location as the block handler would go into in an iseq method frame.

Co-authored-by: John Hawthorn <john@hawthorn.email>

Co-authored-by: John Hawthorn <john@hawthorn.email>
2022-10-04 22:48:05 -04:00
Nobuyoshi Nakada 88c12a2937
Indent folded bootstraptest dots 2022-10-01 18:53:41 +09:00
Aaron Patterson 9a6803c90b
Revert "This commit implements the Object Shapes technique in CRuby."
This reverts commit 68bc9e2e97d12f80df0d113e284864e225f771c2.
2022-09-30 16:01:50 -07:00
Nobuyoshi Nakada 942066713b
bootstraptest/runner: run in parallel if in parallel build 2022-09-29 14:09:25 +09:00
Jemma Issroff d594a5a8bd
This commit implements the Object Shapes technique in CRuby.
Object Shapes is used for accessing instance variables and representing the
"frozenness" of objects.  Object instances have a "shape" and the shape
represents some attributes of the object (currently which instance variables are
set and the "frozenness").  Shapes form a tree data structure, and when a new
instance variable is set on an object, that object "transitions" to a new shape
in the shape tree.  Each shape has an ID that is used for caching. The shape
structure is independent of class, so objects of different types can have the
same shape.

For example:

```ruby
class Foo
  def initialize
    # Starts with shape id 0
    @a = 1 # transitions to shape id 1
    @b = 1 # transitions to shape id 2
  end
end

class Bar
  def initialize
    # Starts with shape id 0
    @a = 1 # transitions to shape id 1
    @b = 1 # transitions to shape id 2
  end
end

foo = Foo.new # `foo` has shape id 2
bar = Bar.new # `bar` has shape id 2
```

Both `foo` and `bar` instances have the same shape because they both set
instance variables of the same name in the same order.

This technique can help to improve inline cache hits as well as generate more
efficient machine code in JIT compilers.

This commit also adds some methods for debugging shapes on objects.  See
`RubyVM::Shape` for more details.

For more context on Object Shapes, see [Feature: #18776]

Co-Authored-By: Aaron Patterson <tenderlove@ruby-lang.org>
Co-Authored-By: Eileen M. Uchitelle <eileencodes@gmail.com>
Co-Authored-By: John Hawthorn <john@hawthorn.email>
2022-09-28 08:26:21 -07:00
Nobuyoshi Nakada 09bce061af bootstraptest/runner: manage sub processes with the job server 2022-09-28 23:18:55 +09:00
Nobuyoshi Nakada fc54dbe8b4 bootstraptest/runner: fold dots by window size 2022-09-28 23:18:55 +09:00
Aaron Patterson 06abfa5be6
Revert this until we can figure out WB issues or remove shapes from GC
Revert "* expand tabs. [ci skip]"

This reverts commit 830b5b5c35.

Revert "This commit implements the Object Shapes technique in CRuby."

This reverts commit 9ddfd2ca00.
2022-09-26 16:10:11 -07:00
Jemma Issroff 9ddfd2ca00 This commit implements the Object Shapes technique in CRuby.
Object Shapes is used for accessing instance variables and representing the
"frozenness" of objects.  Object instances have a "shape" and the shape
represents some attributes of the object (currently which instance variables are
set and the "frozenness").  Shapes form a tree data structure, and when a new
instance variable is set on an object, that object "transitions" to a new shape
in the shape tree.  Each shape has an ID that is used for caching. The shape
structure is independent of class, so objects of different types can have the
same shape.

For example:

```ruby
class Foo
  def initialize
    # Starts with shape id 0
    @a = 1 # transitions to shape id 1
    @b = 1 # transitions to shape id 2
  end
end

class Bar
  def initialize
    # Starts with shape id 0
    @a = 1 # transitions to shape id 1
    @b = 1 # transitions to shape id 2
  end
end

foo = Foo.new # `foo` has shape id 2
bar = Bar.new # `bar` has shape id 2
```

Both `foo` and `bar` instances have the same shape because they both set
instance variables of the same name in the same order.

This technique can help to improve inline cache hits as well as generate more
efficient machine code in JIT compilers.

This commit also adds some methods for debugging shapes on objects.  See
`RubyVM::Shape` for more details.

For more context on Object Shapes, see [Feature: #18776]

Co-Authored-By: Aaron Patterson <tenderlove@ruby-lang.org>
Co-Authored-By: Eileen M. Uchitelle <eileencodes@gmail.com>
Co-Authored-By: John Hawthorn <john@hawthorn.email>
2022-09-26 09:21:30 -07:00
Nobuyoshi Nakada 12e5e5b573
Fix the option name in the error message [ci skip] 2022-09-19 14:15:13 +09:00
John Hawthorn f98d6d3f38
YJIT: Implement specialized respond_to? (#6363)
* Add rb_callable_method_entry_or_negative

* YJIT: Implement specialized respond_to?

This implements a specialized respond_to? in YJIT.

* Update yjit/src/codegen.rs

Co-authored-by: Maxime Chevalier-Boisvert <maximechevalierb@gmail.com>
2022-09-14 16:15:55 -04:00
Hiroshi SHIBATA 29bc3e0daf
omit random failure in bootstraptest with freebsd
http://rubyci.s3.amazonaws.com/freebsd13/ruby-master/log/20220902T063002Z.fail.html.gz
2022-09-06 16:27:12 +09:00
Maple Ong 5a76a15a0f
YJIT: Implement concatarray in yjit (https://github.com/Shopify/ruby/pull/405)
* Create code generation func

* Make rb_vm_concat_array available to use in Rust

* Map opcode to code gen func

* Implement code gen for concatarray

* Add test for concatarray

* Use new asm backend

* Add comment to C func wrapper
2022-08-29 08:47:11 -07:00
Alan Wu 4d811d7a2b
Fix code invalidation while OOM and OOM simulation (https://github.com/Shopify/ruby/pull/395)
`YJIT.simulate_oom!` used to leave one byte of space in the code block,
so our test didn't expose a problem with asserting that the write
position is in bounds in `CodeBlock::set_pos`. We do the following when
patching code:
  1. save current write position
  2. seek to middle of the code block and patch
  3. restore old write position
The bounds check fails on (3) when the code block is already filled up.

Leaving one byte of space also meant that when we write that byte, we
need to fill the entire code region with trapping instruction in
`VirtualMem`, which made the OOM tests unnecessarily slow.

Remove the incorrect bounds check and stop leaving space in the code
block when simulating OOM.
2022-08-29 08:47:10 -07:00
Takashi Kokubun df84832c75
Port getblockparamproxy and getblockparam (https://github.com/Shopify/ruby/pull/394) 2022-08-29 08:47:10 -07:00
Takashi Kokubun ca2afba4a7
Port the remaining method types in opt_send_without_block (https://github.com/Shopify/ruby/pull/390) 2022-08-29 08:47:09 -07:00
Takashi Kokubun 668b99b43b
Port gen_send_iseq to the new backend IR (https://github.com/Shopify/ruby/pull/381)
* Port gen_send_iseq to the new backend IR

* Replace occurrences of 8 by SIZEOF_VALUE

Co-authored-by: Maxime Chevalier-Boisvert <maximechevalierb@gmail.com>
Co-authored-by: Maxime Chevalier-Boisvert <maxime.chevalierboisvert@shopify.com>
2022-08-29 08:47:09 -07:00
Takashi Kokubun 726a451955
Port invokebuiltin* insns to the new backend IR (https://github.com/Shopify/ruby/pull/375)
* Port invokebuiltin* insns to the new backend IR

* Fix the C_ARG_OPNDS check boundary
2022-08-29 08:47:08 -07:00
Takashi Kokubun 2b85295d28
Port objtostring to the new backend (https://github.com/Shopify/ruby/pull/369) 2022-08-29 08:47:07 -07:00
Takashi Kokubun 4539c21367
Port gen_send_cfunc to the new backend (https://github.com/Shopify/ruby/pull/357)
* Port gen_send_cfunc to the new backend

* Remove an obsoleted test

* Add more cfunc tests

* Use csel_e instead and more into()

Co-authored-by: Maxime Chevalier-Boisvert <maximechevalierb@gmail.com>

* Add a missing lea for build_kwargs

* Split cfunc test cases

Co-authored-by: Maxime Chevalier-Boisvert <maximechevalierb@gmail.com>
2022-08-29 08:47:06 -07:00
Zack Deveau dea4238544
Port gen_concatstring to new backend IR (https://github.com/Shopify/ruby/pull/350)
* Port gen_concatstring to new backend IR

* Update yjit/src/codegen.rs

Co-authored-by: Maxime Chevalier-Boisvert <maximechevalierb@gmail.com>
2022-08-29 08:47:05 -07:00
Takashi Kokubun 330c9e9850
Port anytostring, intern, and toregexp (https://github.com/Shopify/ruby/pull/348)
* Port anytostring, intern, and toregexp

* Port getspecial to the new backend (#349)

PR: https://github.com/Shopify/ruby/pull/349
2022-08-29 08:47:05 -07:00
Takashi Kokubun 16307adf8f
Port only ATTRSET of opt_send_without_block (https://github.com/Shopify/ruby/pull/351) 2022-08-29 08:47:05 -07:00
Zack Deveau 6ab71a8598
Port gen_checktype to the new IR assembler backend (https://github.com/Shopify/ruby/pull/343) 2022-08-29 08:47:05 -07:00
Matthew Draper ab08a43ec5
YJIT: Teach getblockparamproxy to handle the no-block case without exiting (#6191)
Teach getblockparamproxy to handle the no-block case without exiting

Co-authored-by: John Hawthorn <john@hawthorn.email>

Co-authored-by: John Hawthorn <john@hawthorn.email>
2022-07-28 11:38:07 -04:00
Hiroshi SHIBATA 98e01c9914
Also skip the failing test similar with a343952d19 2022-07-28 19:10:24 +09:00
Matt Valentine-House 6423d32e3b Replace use of double_heap in tests with expand_heap 2022-07-11 09:00:03 -04:00
Noah Gibbs (and/or Benchmark CI) 5da31b62b0 Make sure string-operation assertions happen inside a method to be sure YJIT will JIT them. 2022-06-27 09:26:18 -07:00
Alan Wu 333754ace8 YJIT: Add regression test for local type tracking
The test in [1] was removed because it stopped working when we limited
the power of Kernel#binding in [2]. However, the underlying issue could
still be reproduced using blocks. Add back a regression test.

I tested the test by commenting out the fix from [1].

[1]: 54c91042ed
[2]: 343ea9967e
2022-06-23 20:27:02 -04:00
Nobuyoshi Nakada a343952d19
On FreeBSD omit the test to close a pipe while reading [ci skip] 2022-06-21 09:41:29 +09:00
Noah Gibbs 9ed9cc9852
Add tests for a variety of string-subclass operations (#5999)
This way YJIT has to match CRuby for each of them.
Remove unused string_p() Rust function
2022-06-10 13:52:43 -04:00
Noah Gibbs 1598c9458a
Add special-case code for the String unary plus operator (#5982) 2022-06-07 11:20:57 -04:00
Noah Gibbs 9d18661e1d
Revert incorrect string-guard optimisation. (#5969)
Also add jhawthorn's test to for this bug.
Fix String#to_s invalidation test
2022-06-01 10:22:08 -04:00
Noah Gibbs 50bad7159a
Special-case jit_guard_known_class for strings. This can remove (#5920)
runtime guard-checks for String#to_s, making some blocks too
short to invalidate later. Add NOPs in those cases to reserve space.
2022-05-20 19:39:37 -04:00
Alan Wu f90549cd38 Rust YJIT
In December 2021, we opened an [issue] to solicit feedback regarding the
porting of the YJIT codebase from C99 to Rust. There were some
reservations, but this project was given the go ahead by Ruby core
developers and Matz. Since then, we have successfully completed the port
of YJIT to Rust.

The new Rust version of YJIT has reached parity with the C version, in
that it passes all the CRuby tests, is able to run all of the YJIT
benchmarks, and performs similarly to the C version (because it works
the same way and largely generates the same machine code). We've even
incorporated some design improvements, such as a more fine-grained
constant invalidation mechanism which we expect will make a big
difference in Ruby on Rails applications.

Because we want to be careful, YJIT is guarded behind a configure
option:

```shell
./configure --enable-yjit # Build YJIT in release mode
./configure --enable-yjit=dev # Build YJIT in dev/debug mode
```

By default, YJIT does not get compiled and cargo/rustc is not required.
If YJIT is built in dev mode, then `cargo` is used to fetch development
dependencies, but when building in release, `cargo` is not required,
only `rustc`. At the moment YJIT requires Rust 1.60.0 or newer.

The YJIT command-line options remain mostly unchanged, and more details
about the build process are documented in `doc/yjit/yjit.md`.

The CI tests have been updated and do not take any more resources than
before.

The development history of the Rust port is available at the following
commit for interested parties:
1fd9573d8b

Our hope is that Rust YJIT will be compiled and included as a part of
system packages and compiled binaries of the Ruby 3.2 release. We do not
anticipate any major problems as Rust is well supported on every
platform which YJIT supports, but to make sure that this process works
smoothly, we would like to reach out to those who take care of building
systems packages before the 3.2 release is shipped and resolve any
issues that may come up.

[issue]: https://bugs.ruby-lang.org/issues/18481

Co-authored-by: Maxime Chevalier-Boisvert <maximechevalierb@gmail.com>
Co-authored-by: Noah Gibbs <the.codefolio.guy@gmail.com>
Co-authored-by: Kevin Newton <kddnewton@gmail.com>
2022-04-27 11:00:22 -04:00
Jeremy Evans 0b091fdac6
Raise RuntimeError if Kernel#binding is called from a non-Ruby frame
Check whether the current or previous frame is a Ruby frame in
call_trace_func and rb_tracearg_binding before attempting to
create a binding for the frame.

Fixes [Bug #18487]

Co-authored-by: Alan Wu <XrXr@users.noreply.github.com>
2022-04-06 19:14:03 -07:00
Kevin Newton 6068da8937 Finer-grained constant cache invalidation (take 2)
This commit reintroduces finer-grained constant cache invalidation.
After 8008fb7 got merged, it was causing issues on token-threaded
builds (such as on Windows).

The issue was that when you're iterating through instruction sequences
and using the translator functions to get back the instruction structs,
you're either using `rb_vm_insn_null_translator` or
`rb_vm_insn_addr2insn2` depending if it's a direct-threading build.
`rb_vm_insn_addr2insn2` does some normalization to always return to
you the non-trace version of whatever instruction you're looking at.
`rb_vm_insn_null_translator` does not do that normalization.

This means that when you're looping through the instructions if you're
trying to do an opcode comparison, it can change depending on the type
of threading that you're using. This can be very confusing. So, this
commit creates a new translator function
`rb_vm_insn_normalizing_translator` to always return the non-trace
version so that opcode comparisons don't have to worry about different
configurations.

[Feature #18589]
2022-04-01 14:48:22 -04:00
Jeremy Evans d1d48cb690 Revert "Raise RuntimeError if Kernel#binding is called from a non-Ruby frame"
This reverts commit 343ea9967e.

This causes an assertion failure with -DRUBY_DEBUG=1 -DRGENGC_CHECK_MODE=2
2022-04-01 07:22:49 -07:00
Nobuyoshi Nakada 69967ee64e
Revert "Finer-grained inline constant cache invalidation"
This reverts commits for [Feature #18589]:
* 8008fb7352
  "Update formatting per feedback"
* 8f6eaca2e1
  "Delete ID from constant cache table if it becomes empty on ISEQ free"
* 629908586b
  "Finer-grained inline constant cache invalidation"

MSWin builds on AppVeyor have been crashing since the merger.
2022-03-25 20:29:09 +09:00
Jeremy Evans 343ea9967e Raise RuntimeError if Kernel#binding is called from a non-Ruby frame
Check whether the current or previous frame is a Ruby frame in
call_trace_func before attempting to create a binding for the frame.

Fixes [Bug #18487]

Co-authored-by: Alan Wu <XrXr@users.noreply.github.com>
2022-03-24 12:31:07 -07:00
Kevin Newton 629908586b Finer-grained inline constant cache invalidation
Current behavior - caches depend on a global counter. All constant mutations cause caches to be invalidated.

```ruby
class A
  B = 1
end

def foo
  A::B # inline cache depends on global counter
end

foo # populate inline cache
foo # hit inline cache

C = 1 # global counter increments, all caches are invalidated

foo # misses inline cache due to `C = 1`
```

Proposed behavior - caches depend on name components. Only constant mutations with corresponding names will invalidate the cache.

```ruby
class A
  B = 1
end

def foo
  A::B # inline cache depends constants named "A" and "B"
end

foo # populate inline cache
foo # hit inline cache

C = 1 # caches that depend on the name "C" are invalidated

foo # hits inline cache because IC only depends on "A" and "B"
```

Examples of breaking the new cache:

```ruby
module C
  # Breaks `foo` cache because "A" constant is set and the cache in foo depends
  # on "A" and "B"
  class A; end
end

B = 1
```

We expect the new cache scheme to be invalidated less often because names aren't frequently reused. With the cache being invalidated less, we can rely on its stability more to keep our constant references fast and reduce the need to throw away generated code in YJIT.
2022-03-24 09:14:38 -07:00
Nobuyoshi Nakada c8b414b334
Reuse `-v` option result as `target_platform`
Backticks method invokes `/bin/sh` when the command contains
quotes, and `sh` clears some environment variables set in
runruby.rb to search the built shared library.
2022-02-17 20:12:38 +09:00
Koichi Sasada fdf0f8d814 btest-ruby OPTS=-v should disable quiet
`make btest-ruby` is run with -q (quiet) option and -v should
remove -q option.
2022-02-17 18:58:37 +09:00
Koichi Sasada 8e0899919b extend timeout for mjit
1 seconeds is not enough on an specific (busy) machine w/ mjit.
http://ci.rvm.jp/results/trunk-mjit@phosphorus-docker/3830178
2022-02-14 14:52:12 +09:00
Koichi Sasada 603ab70961 support concurrent btest execution
* `-j` option for concurrent test with threads
  * `-jN` uses N threads
  * `-j` uses nproc/2 threads
* Introduce `BT` struct to manage configurations
* Introduce `Assertion` to manage all assertions
* Remove all toplevel instance variables
* Show elapsed seconds at last

```
$ time make btest
...
real    0m37.319s
user    0m26.221s
sys     0m16.534s

$ time make btest TESTOPTS=-j
...
real    0m11.812s
user    0m36.667s
sys     0m21.872s
```
2022-02-06 03:05:47 +09:00
Koichi Sasada e953d68657 refactoring btest output
don't duplicate the message, but prepare `out` var to choose
stdout/err.
2022-01-26 09:56:27 +09:00
Koichi Sasada 5e3a320218 respect `--quiet` option for btest
Do not print anymore except errors.
2022-01-21 17:38:11 +09:00
Yuta Saito dd99ee1f7d [wasm] bootstraptest, basictest: disable backquote literal tests
WASI doesn't support spawning a new process for now.
2022-01-19 11:19:06 +09:00
John Hawthorn 930ebdd7ca YJIT: Support kwargs for cfunc
This adds support for passing keyword arguments to cfuncs. This is done
by calling a helper method to create the hash from the top N values on
the stack (determined by the callinfo) and then moving that value onto
the stack.
2022-01-10 10:53:31 -08:00
John Hawthorn 18b97eee5a YJIT: Add support for ruby array cfuncs (argc=-2)
This adds support for cfuncs which take variable arguments using a Ruby
array. This is specified with the method entry's argc == -2.
2022-01-08 11:40:55 -08:00
Alan Wu 54c91042ed YJIT: Discard local var type info on routine call
Routines that are called from YJIT's output code can call methods, and
calling methods mean they can capture and change the environment of the
calling frame.

Discard type info whenever we perform routine calls. This is more
conservative than strictly necessary as some routines need to perform GC
allocation but can never call methods and so should never be able to
change local variables. However, manually analyzing C functions for
whether they have code paths that call methods is error prone and can go
out of date as changes land in the codebase.

Closes: shopify/yjit#300
2022-01-07 19:29:03 -05:00
John Hawthorn 5414de4b6e YJIT: Fix SP index with optarg and unordered kwarg
Previously when we were calling a method with an optional argument and
multiple keywords arguments which weren't in the order the receiver
expected we would use the wrong SP index to rearrange them.

Fixes Bug #18453
2021-12-31 17:30:47 -08:00
Koichi Sasada 6050e3e2a6 @@cv is not accessible from non-main ractors
Class variables (@@cv) is not accessible from non-main ractors.
But without this patch cached @@cv can be read.

fix [Bug #18128]
2021-12-24 13:52:02 +09:00
John Hawthorn c2197bf821 YJIT: Fix check for required kwargs
Previously, YJIT would not check that all the required keywords were
specified in the case that there were optional arguments specified. In
this case YJIT would incorrectly call the method with invalid arguments.
2021-12-17 15:26:04 -08:00
John Hawthorn 83aa68447c YJIT: Allow iseq with both opt and kwargs
Previously we mirrored the fast paths the interpreter had for having
only one of kwargs or optional args. This commit aims to combine the
cases and reduce complexity.

Though this allows calling iseqs which have have both optional and
keyword arguments, it requires that all optional arguments are specified
when there are keyword arguments, since unspecified optional arguments
appear before the kwargs. Support for this can be added a in a future
PR.
2021-12-17 15:26:04 -08:00
Yuta Saito 332d1e52e6 btest: assign $stderr = STDOUT instead of IO#reopen to be more portable
`IO#reopen` internally uses dup syscall but some platforms don't support
the syscall. re-assigning `$stderr` is enough to capture the interpreter's
errors and warnings.
2021-12-16 16:16:40 +09:00
Alan Wu ac5d6faea8
YJIT: Fix unexpected truncation when outputing VALUE
Previously, YJIT incorrectly discarded the upper 32 bits of the object
pointer when writing out VALUEs to setup default keyword arguments.

In addition to incorrectly truncating, the output pointers were not
properly tracked for handling GC compaction moving the referenced
objects.

YJIT previously attempted to encode a mov instruction with a memory
destination and a 64 bit immediate when there is no such encoding
possible in the ISA. Add an assert to mitigate not being able to
catch this at build time.
2021-12-14 19:47:42 -05:00
Koichi Sasada 397a509b6d prohibit load by `autoload` on non-main Ractor
fix [Bug #18120]
2021-12-15 02:33:17 +09:00
Takashi Kokubun 1a63468831
Prepare for removing RubyVM::JIT (#5262) 2021-12-13 23:07:46 -08:00
Koichi Sasada cce331272b `Ractor.make_shareable` checks proc's sefl
`Ractor.make_shareable(proc_obj)` raises an `IsolationError`
if the self of `proc_obj` is not a shareable object.

[Bug #18243]
2021-12-09 16:20:04 +09:00
Alan Wu 82bb9cedd3 YJIT: Fix leak in compilation loop
Previously, when there are too many blocks in a batch, the last block in
the batch is not tracked in the array of batches and not freed.
2021-12-08 16:59:52 -05:00
Alan Wu 286c07f0dc YJIT: Remove guard_self_is_heap()
It's superseded by functionality added to jit_guard_known_klass().
In weird situations such as the ones in the included test,
guard_self_is_heap() triggered assertions.

Co-authored-by: Jemma Issroff <jemmaissroff@gmail.com>
2021-12-07 17:20:34 -05:00
Alan Wu b7ea66bc32 YJIT: Fix incomplete invalidation from opt_setinlinecache
As part of YJIT's strategy for promoting Ruby constant expressions into
constants in the output native code, the interpreter calls
rb_yjit_constant_ic_update() from opt_setinlinecache.

The block invalidation loop indirectly calls rb_darray_remove_unordered(),
which does a shuffle remove. Because of this, looping with an
incrementing counter like done previously can miss some elements in the
array. Repeatedly invalidate the first element instead.

The bug this commit resolves does not seem to cause crashes or divergent
behaviors.

Co-authored-by: Jemma Issroff <jemmaissroff@gmail.com>
2021-12-06 19:24:41 -05:00
Alan Wu 34b5e2566d YJIT: Enable out of memory tests
As of [1] and [2], YJIT has enough support for out of memory conditions
to pass these two basic tests.

OOM code paths are prone to bugs since they are rarely exercised in
common workloads. We might want to add CI runs that stress test these
code paths. Maybe outside of GitHub Actions for capacity reasons.

[1]: f41b4d44f9
[2]: b5b6ab4194
2021-12-04 11:35:37 -05:00
John Hawthorn 733500e9d0
Lazily create singletons on instance_{exec,eval} (#5146)
* Lazily create singletons on instance_{exec,eval}

Previously when instance_exec or instance_eval was called on an object,
that object would be given a singleton class so that method
definitions inside the block would be added to the object rather than
its class.

This commit aims to improve performance by delaying the creation of the
singleton class unless/until one is needed for method definition. Most
of the time instance_eval is used without any method definition.

This was implemented by adding a flag to the cref indicating that it
represents a singleton of the object rather than a class itself. In this
case CREF_CLASS returns the object's existing class, but in cases that
we are defining a method (either via definemethod or
VM_SPECIAL_OBJECT_CBASE which is used for undef and alias).

This also happens to fix what I believe is a bug. Previously
instance_eval behaved differently with regards to constant access for
true/false/nil than for all other objects. I don't think this was
intentional.

    String::Foo = "foo"
    "".instance_eval("Foo")   # => "foo"
    Integer::Foo = "foo"
    123.instance_eval("Foo")  # => "foo"
    TrueClass::Foo = "foo"
    true.instance_eval("Foo") # NameError: uninitialized constant Foo

This also slightly changes the error message when trying to define a method
through instance_eval on an object which can't have a singleton class.

Before:

    $ ruby -e '123.instance_eval { def foo; end }'
    -e:1:in `block in <main>': no class/module to add method (TypeError)

After:

    $ ./ruby -e '123.instance_eval { def foo; end }'
    -e:1:in `block in <main>': can't define singleton (TypeError)

IMO this error is a small improvement on the original and better matches
the (both old and new) message when definging a method using `def self.`

    $ ruby -e '123.instance_eval{ def self.foo; end }'
    -e:1:in `block in <main>': can't define singleton (TypeError)

Co-authored-by: Matthew Draper <matthew@trebex.net>

* Remove "under" argument from yield_under

* Move CREF_SINGLETON_SET into vm_cref_new

* Simplify vm_get_const_base

* Fix leaf VM_SPECIAL_OBJECT_CONST_BASE

Co-authored-by: Matthew Draper <matthew@trebex.net>
2021-12-02 15:53:39 -08:00
Alan Wu d0772632bf YJIT: Fail gracefully while OOM for new entry points
Previously, YJIT crashes with rb_bug() when asked to compile new methods
while out of executable memory.

To handle this situation gracefully, this change keeps track of all the
blocks compiled each invocation in case YJIT runs out of memory in the
middle of a compliation sequence. The list is used to free all blocks in
case compilation fails.

yjit_gen_block() is renamed to gen_single_block() to make it distinct from
gen_block_version(). Call to limit_block_version() and block_t
allocation is moved into the function to help tidy error checking in the
outer loop.

limit_block_version() now returns by value. I feel that an out parameter
with conditional mutation is unnecessarily hard to read in code that
does not need to go for last drop performance. There is a good chance
that the optimizer is able to output identical code anyways.
2021-12-01 12:25:28 -05:00
Alan Wu b5b6ab4194
YJIT: Add ability to exit to interpreter from stubs
Previously, YJIT assumed that it's always possible to generate a new
basic block when servicing a stub in branch_stub_hit(). When YJIT is out
of executable memory, for example, this assumption doesn't hold up.

Add handling to branch_stub_hit() for servicing stubs without consuming
more executable memory by adding a code path that exits to the
interpreter at the location the branch stub represents. The new code
path reconstructs interpreter state in branch_stub_hit() and then exits
with a new snippet called `code_for_exit_from_stub` that returns
`Qundef` from the YJIT native stack frame.

As this change adds another place where we regenerate code from
`branch_t`, extract the logic for it into a new function and call it
regenerate_branch(). While we are at it, make the branch shrinking code
path in branch_stub_hit() more explicit.

This new functionality is hard to test without full support for out of
memory conditions. To verify this change, I ran
`RUBY_YJIT_ENABLE=1 make check -j12` with the following patch to stress
test the new code path:

```diff
diff --git a/yjit_core.c b/yjit_core.c
index 4ab63d9806..5788b8c5ed 100644
--- a/yjit_core.c
+++ b/yjit_core.c
@@ -878,8 +878,12 @@ branch_stub_hit(branch_t *branch, const uint32_t target_idx, rb_execution_contex
                 cb_set_write_ptr(cb, branch->end_addr);
             }

+if (rand() < RAND_MAX/2) {
             // Compile the new block version
             p_block = gen_block_version(target, target_ctx, ec);
+}else{
+    p_block = NULL;
+}

             if (!p_block && branch_modified) {
                 // We couldn't generate a new block for the branch, but we modified the branch.
```

We can enable the new test along with other OOM tests once full support
lands.

Other small changes:
 * yjit_utils.c (print_str): Update to work with new native frame shape.
       Follow up for 8fa0ee4d40.
 * yjit_iface.c (rb_yjit_init): Run yjit_init_core() after
       yjit_init_codegen() so `cb` and `ocb` are available.
2021-11-26 18:00:42 -05:00
John Hawthorn de9a1e4a96
YJIT: Implement new struct accessors (#5161)
* YJIT: Implement optimized_method_struct_aref

* YJIT: Implement struct_aref without method call

Struct member reads can be compiled directly into a memory read (with
either one or two levels of indirection).

* YJIT: Implement optimized struct aset

* YJIT: Update tests for struct access

* YJIT: Add counters for remaining optimized methods

* Check for INT32_MAX overflow

It only takes a struct with 0x7fffffff/8+1 members. Also add some
cheap compile time checks.

* Add tests for non-embedded struct aref/aset

Co-authored-by: Alan Wu <XrXr@users.noreply.github.com>
2021-11-25 11:56:58 -08:00
Alan Wu 13d1ded253 YJIT: Make block invalidation more robust
This commit adds an entry_exit field to block_t for use in
invalidate_block_version(). By patching the start of the block while
invalidating it, invalidate_block_version() can function correctly
while there is no executable memory left for new branch stubs.

This change additionally fixes correctness for situations where we
cannot patch incoming jumps to the invalidated block. In situations
such as Shopify/yjit#226, the address to the start of the block
is saved and used later, possibly after the block is invalidated.

The assume_* family of function now generate block->entry_exit before
remembering blocks for invalidation.

RubyVM::YJIT.simulate_oom! is introduced for testing out of memory
conditions. The test for it is disabled for now because OOM triggers
other failure conditions not addressed by this commit.

Fixes Shopify/yjit#226
2021-11-22 18:23:28 -05:00
Adam Hess 73388aff5e
Add YJIT codegen for objtostring (#5149)
This is the minimal correct objtostring implementation in YJIT.
For correctness, it is important that to_string not get called on strings or subclasses of string.
There is a new test for this behavior.

A follow up should implement an optimized version for other types as performed in `vm_objtostring`.

Co-authored-by: John Hawthorn <jhawthorn@github.com>

Co-authored-by: John Hawthorn <jhawthorn@github.com>
2021-11-19 16:57:09 -05:00
Nikita Vasilevsky c1c13c58ee Add one more test example for swap instruction 2021-11-09 16:56:44 +09:00
Nobuyoshi Nakada 334b69e504 rb_id_serial_to_id: return unregistered ID as an internal ID
```ruby
def foo(*); ->{ super }; end
```

This code makes anonymous parameters which is not registered as an
ID.  The problem is that when Ractors try to scan `getlocal`
instructions, it puts the Symbol corresponding to the parameter
in to a hash.  Since it is not registered, we end up with a
strange exception.  This commit wraps the unregistered ID in an
internal ID so that we get the same exception for `...` as `*`.

Co-Authored-By: Aaron Patterson <tenderlove@ruby-lang.org>
Co-Authored-By: John Hawthorn <john@hawthorn.email>
2021-11-07 12:40:27 +09:00
John Hawthorn fbd6cc5856
YJIT: Support iseq sends with mixed kwargs (#5082)
* YJIT: Support iseq sends with mixed kwargs

Co-authored-by: Kevin Newton <kddnewton@gmail.com>

* Add additional comments to iseq sends

Co-authored-by: Kevin Newton <kddnewton@gmail.com>
2021-11-05 17:01:07 -04:00
John Hawthorn 2fa51c7068
YJIT: Support kwargs sends with all defaults (#5067)
* YJIT: Support kwargs sends with all defaults

Previously keyword argument methods were only compiled by YJIT when all
keywords were specified in the caller.

This adds support for calling methods with keyword arguments when no
keyword arguments are specified and all are filled with the defaults.

* Remove unused send_iseq_kwargs_none_passed
2021-11-01 10:54:59 -04:00
John Hawthorn a6104b392a
YJIT: Support newhash with values (#5029)
* YJIT: Implement newhash with values

* YJIT: Add test of duphash

* Fix compilation on macos/clang
2021-10-27 10:55:43 -04:00
Satoshi Moris Tagomori 489e5e3a82 the core problem is the Proc is not shareable 2021-10-27 16:13:43 +09:00
Ian C. Anderson e943511455
YJIT: Implement duphash (#5009)
`duphash` showed up in the top-20 most frequent exit ops for @jhawthorn's benchmark that renders github.com/about

The implementation was almost exactly the same as `duparray`

Co-authored-by: John Hawthorn <john@hawthorn.email>

Co-authored-by: John Hawthorn <john@hawthorn.email>
2021-10-25 10:40:33 -04:00
Koichi Sasada acb23454e5 allow to access ivars of classes/modules
if an ivar of a class/module refer to a shareable object, this ivar
can be read from non-main Ractors.
2021-10-23 01:32:55 +09:00
Alan Wu ba4bf8a1e6
Fix simple test on platforms where compaction is not supported
844588f915 made it so that trying to call
gc_verify_compaction_references on unsupported platform result in an
exception rather than a crash. Rescue the exception in a YJIT btest
that uses gc_verify_compaction_references.
2021-10-22 10:53:42 -04:00
Alan Wu bdfc23cba9
YJIT: don't compile attr_accessor methods when tracing (#4998)
2d98593bf5 made it so that
attr_accessor methods fire C method tracing events.
Previously, we weren't checking for whether we are tracing before
compiling, leading to missed events.

Since global invalidation invalidates all code, and that attr_accessor
methods can never enable tracing while running, events are only dropped
when YJIT tries to compile when tracing is already enabled.

Factor out the code for checking tracing and check it before generating
code for attr_accessor methods.

This change fixes TestSetTraceFunc#test_tracepoint_attr when it's
ran in isolation.
2021-10-21 15:07:32 -04:00
git 7637175a84 * append newline at EOF. [ci skip] 2021-10-21 08:12:53 +09:00