Implementation for Language Server Protocol (LSP) sometimes needs token information.
For example both `m(1)` and `m(1, )` has same AST structure other than node locations
then it's impossible to check the existence of `,` from AST. However in later case,
it might be better to suggest variables list for the second argument.
Token information is important for such case.
This commit adds these methods.
* Add `keep_tokens` option for `RubyVM::AbstractSyntaxTree.parse`, `.parse_file` and `.of`
* Add `RubyVM::AbstractSyntaxTree::Node#tokens` which returns tokens for the node including tokens for descendants nodes.
* Add `RubyVM::AbstractSyntaxTree::Node#all_tokens` which returns all tokens for the input script regardless the receiver node.
[Feature #19070]
Impacts on memory usage and performance are below:
Memory usage:
```
$ cat test.rb
root = RubyVM::AbstractSyntaxTree.parse_file(File.expand_path('../test/ruby/test_keyword.rb', __FILE__), keep_tokens: true)
$ /usr/bin/time -f %Mkb /usr/local/bin/ruby -v
ruby 3.2.0dev (2022-11-19T09:41:54Z 19070-keep_tokens d3af1b8057) [x86_64-linux]
11408kb
# keep_tokens :false
$ /usr/bin/time -f %Mkb /usr/local/bin/ruby test.rb
17508kb
# keep_tokens :true
$ /usr/bin/time -f %Mkb /usr/local/bin/ruby test.rb
30960kb
```
Performance:
```
$ cat ../ast_keep_tokens.yml
prelude: |
src = <<~SRC
module M
class C
def m1(a, b)
1 + a + b
end
end
end
SRC
benchmark:
without_keep_tokens: |
RubyVM::AbstractSyntaxTree.parse(src, keep_tokens: false)
with_keep_tokens: |
RubyVM::AbstractSyntaxTree.parse(src, keep_tokens: true)
$ make benchmark COMPARE_RUBY="./ruby" ARGS=../ast_keep_tokens.yml
/home/kaneko.y/.rbenv/shims/ruby --disable=gems -rrubygems -I../benchmark/lib ../benchmark/benchmark-driver/exe/benchmark-driver \
--executables="compare-ruby::./ruby -I.ext/common --disable-gem" \
--executables="built-ruby::./miniruby -I../lib -I. -I.ext/common ../tool/runruby.rb --extout=.ext -- --disable-gems --disable-gem" \
--output=markdown --output-compare -v ../ast_keep_tokens.yml
compare-ruby: ruby 3.2.0dev (2022-11-19T09:41:54Z 19070-keep_tokens d3af1b8057) [x86_64-linux]
built-ruby: ruby 3.2.0dev (2022-11-19T09:41:54Z 19070-keep_tokens d3af1b8057) [x86_64-linux]
warming up..
| |compare-ruby|built-ruby|
|:--------------------|-----------:|---------:|
|without_keep_tokens | 21.659k| 21.303k|
| | 1.02x| -|
|with_keep_tokens | 6.220k| 5.691k|
| | 1.09x| -|
```
As there should be no modified files just affter `git cherry-pick`
succeeded in `sync_default_gems_with_commits`, reset to the previous
revision once to pick up the committed files.
win32/win32.c: In function 'rtc_error_handler':
win32/win32.c:691:5: warning: function 'rtc_error_handler' might be a candidate for 'gnu_printf' format attribute [-Wsuggest-attribute=format]
691 | rb_str_vcatf(str, fmt, ap);
| ^~~~~~~~~~~~
and
win32/win32.c:683:1: warning: 'rtc_error_handler' defined but not used [-Wunused-function]
683 | rtc_error_handler(int e, const char *src, int line, const char *exe, const char *fmt, ...)
| ^~~~~~~~~~~~~~~~~
The last part of the sentence was accidentally put in enumeration, It
made an impression that it's one of the rules, not the result of
applying the rules.
We use this code path when we require the same extension twice, so it's
not necessarily about the feature being statically linked. Removing this
code when there is no statically linked extensions leads to breakage.
The tests looks for the .so file, which doesn't exist when the extension
is statically linked. In that situation it passes -I to ruby with
nothing after it which ate the -rio/console argument.
(https://github.com/ruby/irb/pull/448)
* Minor fixes on debug command
* Discover and load debug.gem even if it's not in Gemfile
* Eliminate else for rescue
* Discover the latest one from all gem paths
We would like to differentiate types of objects via their shape. This
commit adds a special T_OBJECT shape when we allocate an instance of
T_OBJECT. This allows us to avoid testing whether an object is an
instance of a T_OBJECT or not, we can just check the shape.
http://rubyci.s3.amazonaws.com/ubuntu2004-arm/ruby-master/log/20221118T033003Z.log.html.gz
```
[21350/22024] TestIRB::ExtendCommandTest#test_show_source_end_finder/home/chkbuild/chkbuild/tmp/build/20221118T033003Z/ruby/test/irb/test_cmd.rb:523: warning: method redefined; discarding old show_source_test_method
/home/chkbuild/chkbuild/tmp/build/20221118T033003Z/ruby/test/irb/test_cmd.rb:523: warning: previous definition of show_source_test_method was here
```
This reverts commit bd148a2bdd.
ERB#result_with_hash does not work on Ruby 2.2
https://ci.appveyor.com/project/ruby/ruby/builds/45420170
```
../tool/generic_erb.rb:33:in `block (2 levels) in <main>': undefined method `result_with_hash' for #<ERB:0x0000000002516650> (NoMethodError)
```
@casperisfine reporting a bug in this gist https://gist.github.com/casperisfine/d59e297fba38eb3905a3d7152b9e9350
After investigating I found it was caused by a combination of send and a c_func that we have overwritten in the JIT. For send calls, we need to do some stack manipulation before making the call. Because of the way exits works, we need to do that stack manipulation at the last possible moment. In this case, we weren't doing that stack manipulation at all. Unfortunately, with how the code is structured there isn't a great place to do that stack manipulation for our overridden C funcs.
Each overridden C func can return a boolean stating that it shouldn't be used. We would need to do the stack manipulation after all of those checks are done. We could pass a lambda(?) or separate out the logic for "can I run this override" from "now generate the code for it". Since we are coming up on a release, I went with the path of least resistence and just decided to not use these overrides if we are in a send call.
We definitely should revist this in the future.
Since https://github.com/ruby/ruby/pull/6599, RUBY_IMMEDIATE_MASK also
overlaps RUBY_Qnil. Now RB_SPECIAL_CONST_P seems confusing since both
RB_IMMEDIATE_P and RB_TEST check for RUBY_Qnil while we only need to
check RUBY_Qnil besides RUBY_IMMEDIATE_MASK. I'd like to make this
change to make it less confusing.
I confirmed that this doesn't change the number of instructions used for
the RUBY_Qfalse check on Linux x86_64 GCC and macOS arm64 Clang.
* Windows: Fix warning about undefined if_indextoname()
* Windows: Fix UNIXSocket on MINGW and make .pair more reliable
* Windows: Use nonblock=true for read tests with scheduler
* Windows: Move socket detection from File.socket? to File.stat
Add S_IFSOCK to Windows and interpret reparse points accordingly.
Enable tests that work now.
* Windows: Use wide-char functions to UNIXSocket
This fixes behaviour with non-ASCII characters.
It also fixes deletion of temporary UNIXSocket.pair files.
* Windows: Add UNIXSocket tests for specifics of Windows impl.
* Windows: fix VC build due to missing _snwprintf
Avoid usage of _snwprintf, since it fails linking ruby.dll like so:
linking shared-library x64-vcruntime140-ruby320.dll
x64-vcruntime140-ruby320.def : error LNK2001: unresolved external symbol snwprintf
x64-vcruntime140-ruby320.def : error LNK2001: unresolved external symbol vsnwprintf_l
whereas linking miniruby.exe succeeds.
This patch uses snprintf on the UTF-8 string instead.
Also remove branch GetWindowsDirectoryW, since it doesn't work.
* Windows: Fix dangling symlink test failures
Co-authored-by: Lars Kanis <kanis@comcard.de>
The first `add_pages` call shouldn't hit Code GC yet; otherwise
`compiles` returns false. With the increased code size in runtime_stats
itself, it sometimes hits Code GC too early, at least in arm64 that has
a large code size.
yui-knk
Takashi Kokubun
Nobuyoshi Nakada
Stan Lo
Lars Kanis
BurdetteLamar
Maciek Rząsa
Alan Wu
Alan Wu
Aaron Patterson
Shugo Maeda
Stan Lo
Hiroshi SHIBATA
Yusuke Endoh
Jimmy Miller
Samuel Williams