* YJIT: Reorder branches for Fixnum opt_case_dispatch
Co-authored-by: Maxime Chevalier-Boisvert <maxime.chevalierboisvert@shopify.com>
Co-authored-by: Alan Wu <alansi.xingwu@shopify.com>
* YJIT: Don't support too large values
Co-authored-by: Maxime Chevalier-Boisvert <maxime.chevalierboisvert@shopify.com>
Co-authored-by: Alan Wu <alansi.xingwu@shopify.com>
Calling `String#scan` without a block creates an incomplete MatchData
object whose `RMATCH(match)->str` is Qfalse. Usually this object is not
leaked, but it was possible to pull it by using ObjectSpace.each_object.
This change hides the internal MatchData object by using rb_obj_hide.
Fixes [Bug #19159]
```
$ make test-all TESTS=test/optparse/
...
[148/178] TestOptionParserDidYouMean#test_raise_unknown = 0.00 s
1) Failure:
TestOptionParserDidYouMean#test_raise_unknown [/home/mame/work/ruby/test/optparse/test_optparse.rb:106]:
<["--bar"]> expected but was
<[]>.
```
In the old test/unit (bundled in ruby/ruby), when a test class inherits from
another test class, the child class runs all the tests defined in the parent
class.
However, it looks like the new test/unit does not do so. This is because the
test failure does not occur in ruby/optparse.
As a tentative solution, this changes the option names in TestOptionParser to
avoid the name conflict with TestOptionParserDidYouMean.
https://github.com/ruby/optparse/commit/fee86ef7a4
Most tests in test_yjit.rb use a sub process, so we can run them even
when the parent process is not running with YJIT. Run them so simply
running `make check` tests YJIT a bit.
[Misc #19149]
Previously, for statically-linked extensions, we used
`vm->loading_table` to delay calling the init function until the
extensions are required. This caused the extensions to look like they
are in the middle of being loaded even before they're required.
(`rb_feature_p()` returned true with a loading path output.) Combined
with autoload, queries like `defined?(CONST)` and `Module#autoload?`
were confused by this and returned nil incorrectly. RubyGems uses
`defined?` to detect if OpenSSL is available and failed when OpenSSL was
available in builds using `--with-static-linked-ext`.
Use a dedicated table for the init functions instead of adding them to
the loading table. This lets us remove some logic from non-EXTSTATIC
builds.
[Bug #19115]
It's questionable whether we want to allow rstrip to work for strings
where the broken coderange occurs before the trailing whitespace and
not after, but this approach is probably simpler, and I don't think
users should expect string operations like rstrip to work on broken
strings.
In some cases, this changes rstrip to raise
Encoding::CompatibilityError instead of ArgumentError. However, as
the problem is related to an encoding issue in the receiver, and due
not due to an issue with an argument, I think
Encoding::CompatibilityError is the more appropriate error.
Fixes [Bug #18931]
Beginless ranges previously raised TypeError for this case,
except for string ranges, which had unexpected behavior:
('a'..'z').include?('ww') # false
(..'z').include?('ww') # previously true, now TypeError
Use of include? with endless ranges could previously result
in an infinite loop.
This splits off a range_string_cover_internal function from
range_include_internal.
Fixes [Bug #18580]
We attempted to remove things like this and reverted all of them, but I
think nobody relies on this unlike the positional arguments of
`ERB#initialize`.
https://github.com/ruby/erb/commit/92fde7e403
(https://github.com/ruby/erb/pull/34)
There seems to be no way to fix this. It doesn't happen in an installed
directory, so it's likely a false positive and/or a bug in rubygems.
| <internal:/home/runner/.rubies/ruby-head/lib/ruby/3.2.0+3/rubygems/core_ext/kernel_require.rb>:85: warning: <internal:/home/runner/.rubies/ruby-head/lib/ruby/3.2.0+3/rubygems/core_ext/kernel_require.rb>:85: warning: loading in progress, circular require considered harmful - /home/runner/.rubies/ruby-head/lib/ruby/3.2.0+3/bundler/setup.rb
| from /home/runner/work/erb/erb/libexec/erb:0:in `require'
| from /home/runner/.rubies/ruby-head/lib/ruby/3.2.0+3/bundler/setup.rb:3:in `<top (required)>'
| from /home/runner/.rubies/ruby-head/lib/ruby/3.2.0+3/bundler/setup.rb:3:in `require_relative'
| from /home/runner/.rubies/ruby-head/lib/ruby/3.2.0+3/bundler/shared_helpers.rb:8:in `<top (required)>'
| from /home/runner/.rubies/ruby-head/lib/ruby/3.2.0+3/bundler/shared_helpers.rb:8:in `require_relative'
| from /home/runner/.rubies/ruby-head/lib/ruby/3.2.0+3/bundler/rubygems_integration.rb:3:in `<top (required)>'
| from /home/runner/.rubies/ruby-head/lib/ruby/3.2.0+3/bundler/rubygems_integration.rb:3:in `require'
| from /home/runner/.rubies/ruby-head/lib/ruby/3.2.0+3/rubygems.rb:1352:in `<top (required)>'
| from <internal:/home/runner/.rubies/ruby-head/lib/ruby/3.2.0+3/rubygems/core_ext/kernel_require.rb>:85:in `require'
| from <internal:/home/runner/.rubies/ruby-head/lib/ruby/3.2.0+3/rubygems/core_ext/kernel_require.rb>:85:in `require'
This commit adds RVALUE_OVERHEAD for storing metadata at the end of the
slot. This commit moves the ractor_belonging_id in debug builds from the
flags to RVALUE_OVERHEAD which frees the 16 bits in the headers for
object shapes.
Raise a `SyntaxError` with the parser error message, in the case
reading from a file instead of the `-e` option or standard input. So
syntax_suggest can get the message from the caught error.
(https://github.com/ruby/irb/pull/449)
* Seamlessly integrate a few debug commands
* Improve the break command support
* Utilize skip_src option if available
* Add step and delete commands
* Write end-to-end tests for each debugger command
* Add documentation
* Add backtrace, info, catch commands
https://github.com/ruby/irb/commit/976100c1c2
Implementation for Language Server Protocol (LSP) sometimes needs token information.
For example both `m(1)` and `m(1, )` has same AST structure other than node locations
then it's impossible to check the existence of `,` from AST. However in later case,
it might be better to suggest variables list for the second argument.
Token information is important for such case.
This commit adds these methods.
* Add `keep_tokens` option for `RubyVM::AbstractSyntaxTree.parse`, `.parse_file` and `.of`
* Add `RubyVM::AbstractSyntaxTree::Node#tokens` which returns tokens for the node including tokens for descendants nodes.
* Add `RubyVM::AbstractSyntaxTree::Node#all_tokens` which returns all tokens for the input script regardless the receiver node.
[Feature #19070]
Impacts on memory usage and performance are below:
Memory usage:
```
$ cat test.rb
root = RubyVM::AbstractSyntaxTree.parse_file(File.expand_path('../test/ruby/test_keyword.rb', __FILE__), keep_tokens: true)
$ /usr/bin/time -f %Mkb /usr/local/bin/ruby -v
ruby 3.2.0dev (2022-11-19T09:41:54Z 19070-keep_tokens d3af1b8057) [x86_64-linux]
11408kb
# keep_tokens :false
$ /usr/bin/time -f %Mkb /usr/local/bin/ruby test.rb
17508kb
# keep_tokens :true
$ /usr/bin/time -f %Mkb /usr/local/bin/ruby test.rb
30960kb
```
Performance:
```
$ cat ../ast_keep_tokens.yml
prelude: |
src = <<~SRC
module M
class C
def m1(a, b)
1 + a + b
end
end
end
SRC
benchmark:
without_keep_tokens: |
RubyVM::AbstractSyntaxTree.parse(src, keep_tokens: false)
with_keep_tokens: |
RubyVM::AbstractSyntaxTree.parse(src, keep_tokens: true)
$ make benchmark COMPARE_RUBY="./ruby" ARGS=../ast_keep_tokens.yml
/home/kaneko.y/.rbenv/shims/ruby --disable=gems -rrubygems -I../benchmark/lib ../benchmark/benchmark-driver/exe/benchmark-driver \
--executables="compare-ruby::./ruby -I.ext/common --disable-gem" \
--executables="built-ruby::./miniruby -I../lib -I. -I.ext/common ../tool/runruby.rb --extout=.ext -- --disable-gems --disable-gem" \
--output=markdown --output-compare -v ../ast_keep_tokens.yml
compare-ruby: ruby 3.2.0dev (2022-11-19T09:41:54Z 19070-keep_tokens d3af1b8057) [x86_64-linux]
built-ruby: ruby 3.2.0dev (2022-11-19T09:41:54Z 19070-keep_tokens d3af1b8057) [x86_64-linux]
warming up..
| |compare-ruby|built-ruby|
|:--------------------|-----------:|---------:|
|without_keep_tokens | 21.659k| 21.303k|
| | 1.02x| -|
|with_keep_tokens | 6.220k| 5.691k|
| | 1.09x| -|
```
The tests looks for the .so file, which doesn't exist when the extension
is statically linked. In that situation it passes -I to ruby with
nothing after it which ate the -rio/console argument.
We would like to differentiate types of objects via their shape. This
commit adds a special T_OBJECT shape when we allocate an instance of
T_OBJECT. This allows us to avoid testing whether an object is an
instance of a T_OBJECT or not, we can just check the shape.
http://rubyci.s3.amazonaws.com/ubuntu2004-arm/ruby-master/log/20221118T033003Z.log.html.gz
```
[21350/22024] TestIRB::ExtendCommandTest#test_show_source_end_finder/home/chkbuild/chkbuild/tmp/build/20221118T033003Z/ruby/test/irb/test_cmd.rb:523: warning: method redefined; discarding old show_source_test_method
/home/chkbuild/chkbuild/tmp/build/20221118T033003Z/ruby/test/irb/test_cmd.rb:523: warning: previous definition of show_source_test_method was here
```
* Windows: Fix warning about undefined if_indextoname()
* Windows: Fix UNIXSocket on MINGW and make .pair more reliable
* Windows: Use nonblock=true for read tests with scheduler
* Windows: Move socket detection from File.socket? to File.stat
Add S_IFSOCK to Windows and interpret reparse points accordingly.
Enable tests that work now.
* Windows: Use wide-char functions to UNIXSocket
This fixes behaviour with non-ASCII characters.
It also fixes deletion of temporary UNIXSocket.pair files.
* Windows: Add UNIXSocket tests for specifics of Windows impl.
* Windows: fix VC build due to missing _snwprintf
Avoid usage of _snwprintf, since it fails linking ruby.dll like so:
linking shared-library x64-vcruntime140-ruby320.dll
x64-vcruntime140-ruby320.def : error LNK2001: unresolved external symbol snwprintf
x64-vcruntime140-ruby320.def : error LNK2001: unresolved external symbol vsnwprintf_l
whereas linking miniruby.exe succeeds.
This patch uses snprintf on the UTF-8 string instead.
Also remove branch GetWindowsDirectoryW, since it doesn't work.
* Windows: Fix dangling symlink test failures
Co-authored-by: Lars Kanis <kanis@comcard.de>
The first `add_pages` call shouldn't hit Code GC yet; otherwise
`compiles` returns false. With the increased code size in runtime_stats
itself, it sometimes hits Code GC too early, at least in arm64 that has
a large code size.
(https://github.com/ruby/irb/pull/442)
* Add test_in_isolation task to run tests in isolation
This simulates how tests are run in Ruby CI and can detect some issues
before they're merged and break Ruby CI later.
* Run test_in_isolation task on CI
* Fix TestRaiseNoBacktraceException's setup
https://github.com/ruby/irb/commit/51f23c58b0
The tests failed on windows
https://github.com/ruby/ruby/actions/runs/3440997073/jobs/5740085169#step:18:62
```
1) Failure:
TestRegexp#test_s_timeout [D:/a/ruby/ruby/src/test/ruby/test_regexp.rb:1586]:
<0.30000000000000004> expected but was
<0.3>.
2) Failure:
TestRegexp#test_timeout_shorter_than_global [D:/a/ruby/ruby/src/test/ruby/test_regexp.rb:1631]:
<0.30000000000000004> expected but was
<0.3>.
```
(https://github.com/ruby/irb/pull/437)
* Transform ls's --grep/-G option to keyword args
* Make --grep less flexible
* Support -g instead of --grep
* Suppress warnings from symbol aliases
JRuby uses these same files for testing racc. The existing logic will not
find 'racc' in a JRuby project checkout. This change allows it to work by
just assuming 'ruby -S racc' when running tests. This will not change C
Ruby's detection when setting up tests (since earlier checks will find
racc).
https://github.com/ruby/racc/commit/b74ce5e018
This commit adds a `capacity` field to shapes, and adds shape
transitions whenever an object's capacity changes. Objects which are
allocated out of a bigger size pool will also make a transition from the
root shape to the shape with the correct capacity for their size pool
when they are allocated.
This commit will allow us to remove numiv from objects completely, and
will also mean we can guarantee that if two objects share shapes, their
IVs are in the same positions (an embedded and extended object cannot
share shapes). This will enable us to implement ivar sets in YJIT using
object shapes.
Co-Authored-By: Aaron Patterson <tenderlove@ruby-lang.org>
There is currently a bug ([#19084]) in how extended strings work in
IO Buffers. Object Shapes changes will make the string in this test
extended on 32 bit machines. Since this behavior is currently broken
(unrelated to object shapes) on 32 bit machines, this test will then
fail. We preemptively omit it so that this commit can be reverted
once the bug is fixed.
Nobuyoshi Nakada
Takashi Kokubun
Samuel Williams
Alan Wu
Yusuke Endoh
yui-knk
Shugo Maeda
Sven Riedel
Xenor Chang
nick evans
Stan Lo
Peter Zhu
Jeremy Evans
Jean Boussier
zverok
Aaron Patterson
st0012
Hiroshi SHIBATA
Kenta Murata
Josef Šimánek
Jason Karns
Thomas E. Enebo
Jemma Issroff
TSUYUSATO Kitsune