* So deprecated methods/constants/functions are dealt with early,
instead of many tests breaking suddenly when removing a deprecated
method/constant/function.
* Follows https://bugs.ruby-lang.org/issues/17591
I want to use more complicated macros with MJIT. For example:
```
# define SHAPE_MASK (((unsigned int)1 << SHAPE_BITS) - 1)
```
This commit adds a simple recursive descent parser that produces an AST
and a small visitor that converts the AST to Ruby.
Fixes id.h error during updating ripper.c by `make after-update`.
While it used to update id.h in the build directory, but was trying to
update ripper.c in the source directory. In principle, files in the
source directory can or should not depend on files in the build
directory.
Previously YARV bytecode implemented constant caching by having a pair
of instructions, opt_getinlinecache and opt_setinlinecache, wrapping a
series of getconstant calls (with putobject providing supporting
arguments).
This commit replaces that pattern with a new instruction,
opt_getconstant_path, handling both getting/setting the inline cache and
fetching the constant on a cache miss.
This is implemented by storing the full constant path as a
null-terminated array of IDs inside of the IC structure. idNULL is used
to signal an absolute constant reference.
$ ./miniruby --dump=insns -e '::Foo::Bar::Baz'
== disasm: #<ISeq:<main>@-e:1 (1,0)-(1,13)> (catch: FALSE)
0000 opt_getconstant_path <ic:0 ::Foo::Bar::Baz> ( 1)[Li]
0002 leave
The motivation for this is that we had increasingly found the need to
disassemble the instructions between the opt_getinlinecache and
opt_setinlinecache in order to determine the constant we are fetching,
or otherwise store metadata.
This disassembly was done:
* In opt_setinlinecache, to register the IC against the constant names
it is using for granular invalidation.
* In rb_iseq_free, to unregister the IC from the invalidation table.
* In YJIT to find the position of a opt_getinlinecache instruction to
invalidate it when the cache is populated
* In YJIT to register the constant names being used for invalidation.
With this change we no longe need disassemly for these (in fact
rb_iseq_each is now unused), as the list of constant names being
referenced is held in the IC. This should also make it possible to make
more optimizations in the future.
This may also reduce the size of iseqs, as previously each segment
required 32 bytes (on 64-bit platforms) for each constant segment. This
implementation only stores one ID per-segment.
There should be no significant performance change between this and the
previous implementation. Previously opt_getinlinecache was a "leaf"
instruction, but it included a jump (almost always to a separate cache
line). Now opt_getconstant_path is a non-leaf (it may
raise/autoload/call const_missing) but it does not jump. These seem to
even out.
I'm planning to introduce mjit_compiler.rb, and I want to make this
consistent with it. Consistency with compile.c doesn't seem important
for MJIT anyway.
The current MJIT relies on SIGCHLD and fork(2) to be performant, and
it's something mswin can't offer. You could run Linux MJIT on WSL
instead.
[Misc #18968]
Adds the `syntax_suggest` syntax error display tool to Ruby through the same mechanism as `error_highlight` and `did_you_mean`. Reference ticket: https://bugs.ruby-lang.org/issues/18159close#4845
## What is syntax_suggest?
When a syntax error is raised by requiring a file, dead_end will use a combination of indentation and lexing to identify the problem.
> Note: Previously this tool was named `dead_end`.
## Known issues
- SyntaxSearch's approach of showing syntax errors only works through integration with `require`, `load`, `autoload`, and `require_relative` (since it monkeypatches them to detect syntax errors). It does not work with direct Ruby file invocations https://github.com/zombocom/dead_end/issues/31.
- This causes failure in the test suite (test_expected_backtrace_location_when_inheriting_from_basic_object_and_including_kernel) and confusion when inspecting backtraces if there's a different error when trying to require a file such as measuring memory (https://github.com/zombocom/syntax_suggest/issues/124#issuecomment-1006705016).
- Discussed fix. We previously talked about opening up `SyntaxError` to be monkeypatched in the same way that other gems hook into `NoMethodError`. This is currently not possible and requires development work. When we last talked about it at RubyKaigi Nobu expressed an ability to make such a change.
- Split into `Ext` and `Lib` classes.
- `Ext#files` should not include built extension libraries.
- `Ext#files` should include scripts under its own `lib`.
- `Lib#files` should be prefixed with `lib/`.
Gemspec files generated by old bundler run `git` without changing the
working directory.
Or some gemspec files expect an owned file at the top exists ath the
current working directory.
For the macOS -bundle_loader linker option, we need a path to the
Ruby exectuable. $(RUBY) is not necessarily a path since it could
be a command line invocation. That happens during build with
runruby.rb and can happen post installation if the user passes
the --ruby option to a extconf.rb. Use $(bindir) to locate
the executable instead.
Before installation, $(bindir) doesn't exist, so we need to be
able to override $(BUILTRUBY) in such situations so test-spec
and bundled extensions could build. Use a new mkmf global,
$builtruby, to do this; set it in fake.rb and in extmk.rb.
Our build system is quite complex...
Creates simple bin stubs to load the extracted executable files.
After only extracted under `gems` directory, the gems are considered
installed but the executable scripts are not found.
Also the second argument is now the parent of the previous second and
third arguments.
This commit implements Objects on Variable Width Allocation. This allows
Objects with more ivars to be embedded (i.e. contents directly follow the
object header) which improves performance through better cache locality.
The default gems have not been installed yet in the build directory,
bundled gems depending on them can not work. As those dependencies
should be usable there even without rubygems, make temporary gemspec
files without the dependencies, and use them in the build directory.
Since extension libraries can not be built in the source directory,
rubygems warns gems have extension libraries as the extensions are not
built. To order to suppress this warnings, extract such gemspec files
under each gem directories instead of the common `specifications`
directory.
Pointing `Gem.ruby` to the newly installed ruby gives mkmf the right
inputs to build extensions in bundled gems. Previously, this patching
was only done for compressed bundled gems.
This patch also prevents `tool/fake.rb` from propagating to the child
process running mkmf for the native extension. The way `tool/fake.rb`
changes mkmf variables using `Kernel#trace_var` created spooky action at
a distance which made debugging difficult.
AppVeyor Windows CI started to fail starting with
8a3663789c because it enabled extension
building for bundled gems on mswin. This patch should address the CI
failures.
This commit reintroduces finer-grained constant cache invalidation.
After 8008fb7 got merged, it was causing issues on token-threaded
builds (such as on Windows).
The issue was that when you're iterating through instruction sequences
and using the translator functions to get back the instruction structs,
you're either using `rb_vm_insn_null_translator` or
`rb_vm_insn_addr2insn2` depending if it's a direct-threading build.
`rb_vm_insn_addr2insn2` does some normalization to always return to
you the non-trace version of whatever instruction you're looking at.
`rb_vm_insn_null_translator` does not do that normalization.
This means that when you're looping through the instructions if you're
trying to do an opcode comparison, it can change depending on the type
of threading that you're using. This can be very confusing. So, this
commit creates a new translator function
`rb_vm_insn_normalizing_translator` to always return the non-trace
version so that opcode comparisons don't have to worry about different
configurations.
[Feature #18589]
This reverts commits for [Feature #18589]:
* 8008fb7352
"Update formatting per feedback"
* 8f6eaca2e1
"Delete ID from constant cache table if it becomes empty on ISEQ free"
* 629908586b
"Finer-grained inline constant cache invalidation"
MSWin builds on AppVeyor have been crashing since the merger.
Current behavior - caches depend on a global counter. All constant mutations cause caches to be invalidated.
```ruby
class A
B = 1
end
def foo
A::B # inline cache depends on global counter
end
foo # populate inline cache
foo # hit inline cache
C = 1 # global counter increments, all caches are invalidated
foo # misses inline cache due to `C = 1`
```
Proposed behavior - caches depend on name components. Only constant mutations with corresponding names will invalidate the cache.
```ruby
class A
B = 1
end
def foo
A::B # inline cache depends constants named "A" and "B"
end
foo # populate inline cache
foo # hit inline cache
C = 1 # caches that depend on the name "C" are invalidated
foo # hits inline cache because IC only depends on "A" and "B"
```
Examples of breaking the new cache:
```ruby
module C
# Breaks `foo` cache because "A" constant is set and the cache in foo depends
# on "A" and "B"
class A; end
end
B = 1
```
We expect the new cache scheme to be invalidated less often because names aren't frequently reused. With the cache being invalidated less, we can rely on its stability more to keep our constant references fast and reduce the need to throw away generated code in YJIT.
Use ISEQ_BODY macro to get the rb_iseq_constant_body of the ISeq. Using
this macro will make it easier for us to change the allocation strategy
of rb_iseq_constant_body when using Variable Width Allocation.
For the case using this script as a library.
- `ExtLibs#process` reads and processes an extlibs file.
- `ExtLibs#process_under` processes all extlibs files under the
given directory.
- `Extlibs.run` parses `ARGV` and lets an instance process the
directories.
Header file include/ruby/internal/abi.h contains RUBY_ABI_VERSION which
is the ABI version. This value should be bumped whenever an ABI
incompatible change is introduced.
When loading dynamic libraries, Ruby will compare its own
`ruby_abi_version` and the `ruby_abi_version` of the loaded library. If
these two values don't match it will raise a `LoadError`. This feature
can also be turned off by setting the environment variable
`RUBY_RUBY_ABI_CHECK=0`.
This feature will prevent cases where previously installed native gems
fail in unexpected ways due to incompatibility of changes in header
files. This will force the developer to recompile their gems to use the
same header files as the built Ruby.
In Ruby, the ABI version is exposed through
`RbConfig::CONFIG["ruby_abi_version"]`.
On the parallel test, workers can be killed because of timeout
and the information for the retrying can be inconsistent.
This patch will skip if the inconsistency is found and report
as an error.
http://ci.rvm.jp/results/trunk-asserts@phosphorus-docker/3834082
Issue only occurs in JRuby 9.3.0.0 and Windows and the full
console output is:
log rotation inter-process lock failed. D:\log.txt -> D:\log.txt.0: The process cannot access the file because it is being used by another process.
log writing failed. closed stream
log writing failed. closed stream
...
https://github.com/ruby/logger/commit/19fc734638
The gem doesn't even install on old rubies, but since the gemspec claims
it's supported, `gem install pathname` will try to install it and print
an error.
This commit doesn't fix the above issue. The only way to fix it would be
to restore support and release a new version that actually supports old
rubies. However, such a change has been proposed and ignored for a long
time.
So this issue proposes to leave that broken but at least bring the
gemspec manifest and the CI matrix in sync to hopefully avoid this issue
from happening again in the future.
https://github.com/ruby/pathname/commit/3ee010b538
configure.ac: setup build tools and register objects
main.c: wrap main with rb_wasm_rt_start to handle asyncify unwinds
tool/m4/ruby_wasm_tools.m4: setup default command based on WASI_SDK_PATH
environment variable. checks wasm-opt which is used for asyncify.
tool/wasm-clangw wasm/wasm-opt: a clang wrapper which replaces real
wasm-opt with do-nothing wasm-opt to avoid misoptimization before
asyncify. asyncify is performed at POSTLINK, but clang linker driver
tries to run optimization by wasm-opt unconditionally. inlining pass
at wasm level breaks asyncify's assumption, so should not optimize
before POSTLIK.
wasm/GNUmakefile.in: wasm specific rules to compile objects
When building with --with-static-linked-ext, some exts without rb file
doesn't produce neither .so or .rb under .ext/common. Therefore, change
rbinstall.rb to install gemspec even if there is no .so or .rb for that
case.
With /Z7, no .pdb file is generated, so trying to link it during build
fails on my machine even though it's okay on CI.
By the way, in my local testing, no .pdb is generated in cwd at runtime
even without the /Fd option. I guess we can pass it just in case.
Compare with the C methods, A built-in methods written in Ruby is
slower if only mandatory parameters are given because it needs to
check the argumens and fill default values for optional and keyword
parameters (C methods can check the number of parameters with `argc`,
so there are no overhead). Passing mandatory arguments are common
(optional arguments are exceptional, in many cases) so it is important
to provide the fast path for such common cases.
`Primitive.mandatory_only?` is a special builtin function used with
`if` expression like that:
```ruby
def self.at(time, subsec = false, unit = :microsecond, in: nil)
if Primitive.mandatory_only?
Primitive.time_s_at1(time)
else
Primitive.time_s_at(time, subsec, unit, Primitive.arg!(:in))
end
end
```
and it makes two ISeq,
```
def self.at(time, subsec = false, unit = :microsecond, in: nil)
Primitive.time_s_at(time, subsec, unit, Primitive.arg!(:in))
end
def self.at(time)
Primitive.time_s_at1(time)
end
```
and (2) is pointed by (1). Note that `Primitive.mandatory_only?`
should be used only in a condition of an `if` statement and the
`if` statement should be equal to the methdo body (you can not
put any expression before and after the `if` statement).
A method entry with `mandatory_only?` (`Time.at` on the above case)
is marked as `iseq_overload`. When the method will be dispatch only
with mandatory arguments (`Time.at(0)` for example), make another
method entry with ISeq (2) as mandatory only method entry and it
will be cached in an inline method cache.
The idea is similar discussed in https://bugs.ruby-lang.org/issues/16254
but it only checks mandatory parameters or more, because many cases
only mandatory parameters are given. If we find other cases (optional
or keyword parameters are used frequently and it hurts performance),
we can extend the feature.
* Bundle RBS 1.7.0
* tool/test-bundled-gems.rb: Use a correct path to Check if rbs is built
* tool/test-bundled-gems.rb: lib/rbs/parse.y is no longer created
Co-authored-by: Yusuke Endoh <mame@ruby-lang.org>
If sole `filter` option doesn't seem including test case name,
match with method name only.
And if the filter is a Regexp or String, it never matches method
name symbols.
zlib.net rarely returns the current time in RFC 2616 noncompliant
format in the response header, and the checksum does not match in
that case (maybe creating the tarball on the fly?).
Lets consider the following scenario:
~~~
irb(#<Test::Unit::AutoRunner::Runner:0x0000560f68afc3c8>):001:0> p suite
OpenSSL::TestEC
=> OpenSSL::TestEC
irb(#<Test::Unit::AutoRunner::Runner:0x0000560f68afc3c8>):002:0> p all_test_methods
["test_ECPrivateKey", "test_ECPrivateKey_encrypted", "test_PUBKEY", "test_check_key", "test_derive_key", "test_dh_compute_key", "test_dsa_sign_asn1_FIPS186_3", "test_ec_group", "test_ec_key", "test_ec_point", "test_ec_point_add", "test_ec_point_mul", "test_generate", "test_marshal", "test_sign_verify", "test_sign_verify_raw"]
=>
["test_ECPrivateKey",
"test_ECPrivateKey_encrypted",
"test_PUBKEY",
"test_check_key",
"test_derive_key",
"test_dh_compute_key",
"test_dsa_sign_asn1_FIPS186_3",
"test_ec_group",
"test_ec_key",
"test_ec_point",
"test_ec_point_add",
"test_ec_point_mul",
"test_generate",
"test_marshal",
"test_sign_verify",
"test_sign_verify_raw"]
irb(#<Test::Unit::AutoRunner::Runner:0x0000560f68afc3c8>):003:0> p filter
/\A(?=.*)(?!.*(?-mix:(?-mix:memory_leak)|(?-mix:OpenSSL::TestEC.test_check_key)))/
=> /\A(?=.*)(?!.*(?-mix:(?-mix:memory_leak)|(?-mix:OpenSSL::TestEC.test_check_key)))/
irb(#<Test::Unit::AutoRunner::Runner:0x0000560f68afc3c8>):004:0> method = "test_check_key"
=> "test_check_key"
~~~
The intention here is to exclude the `test_check_key` test case.
Unfortunately this does not work as expected, because the negative filter
is never checked:
~~~
irb(#<Test::Unit::AutoRunner::Runner:0x0000560f68afc3c8>):005:0> filter === method
=> true
irb(#<Test::Unit::AutoRunner::Runner:0x0000560f68afc3c8>):006:0> filter === "#{suite}##{method}"
=> false
irb(#<Test::Unit::AutoRunner::Runner:0x0000560f68afc3c8>):007:0> filter === method || filter === "#{suite}##{method}"
=> true
~~~
Therefore always filter against the fully qualified method name
`#{suite}##{method}`, which should provide the expected result.
However, if plain string filter is used, keep checking also only the
method name.
This resolves [Bug #16936].
Lazily compile out a chain of checks for different known classes and
whether `self` embeds its ivars or not.
* Remove trailing whitespaces
* Get proper addresss in Capstone disassembly
* Lowercase address in Capstone disassembly
Capstone uses lowercase for jump targets in generated listings. Let's
match it.
* Use the same successor in getivar guard chains
Cuts down on duplication
* Address reviews
* Fix copypasta error
* Add a comment
* Separate exception classes to be rescued or reraised
* Use the filtered backtrace in the failure message
* Raise a new `AssertionFailedError` with the original backtrace
In 2021, Ruby 2.5 and older are EOL.
We can set the default required Ruby version to 2.6.0 to
encourage people to use newer Ruby.
If the command is executed with old Ruby, it falls back to 2.3.0.
It's still possible to create a gem for older Ruby just by changing
two lines of code (one in gemspec and another is in rubocop.yml).
For the `test-bundled-gems`, make `debug.so` with extconf.rb and
`make` command directly because `rake-compiler` assume ruby is
installed (but `test-bundled-gems` can run without installation).
Needs to override Test::Unit::Runner#run, so that RunCount#run
runs which increments @@run_count. Previously it worked because
these methods were inserted between Test::Unit::Runner#run and
MiniTest::Unit#run.
Hiroshi SHIBATA
Nobuyoshi Nakada
Benoit Daloze
Aaron Patterson
Takashi Kokubun
John Hawthorn
schneems
Burdette Lamar
Alan Wu
Peter Zhu
Jemma Issroff
David Rodríguez
Alan Wu
Jun Aruga
Kazuhiro NISHIYAMA
Kevin Newton
NARUSE, Yui
Koichi Sasada
Jesse Chavez
Yuta Saito
Soutaro Matsumoto
Samuel Williams
Yusuke Endoh
Vít Ondruch
Maxime Chevalier-Boisvert
Jose Narvaez
Pavel Rosický
OKURA Masafumi
fedor