Introduce Universal Parser mode for the parser.
This commit includes these changes:
* Introduce `UNIVERSAL_PARSER` macro. All of CRuby related functions
are passed via `struct rb_parser_config_struct` when this macro is enabled.
* Add CI task with 'cppflags=-DUNIVERSAL_PARSER' for ubuntu.
Implementation for Language Server Protocol (LSP) sometimes needs token information.
For example both `m(1)` and `m(1, )` has same AST structure other than node locations
then it's impossible to check the existence of `,` from AST. However in later case,
it might be better to suggest variables list for the second argument.
Token information is important for such case.
This commit adds these methods.
* Add `keep_tokens` option for `RubyVM::AbstractSyntaxTree.parse`, `.parse_file` and `.of`
* Add `RubyVM::AbstractSyntaxTree::Node#tokens` which returns tokens for the node including tokens for descendants nodes.
* Add `RubyVM::AbstractSyntaxTree::Node#all_tokens` which returns all tokens for the input script regardless the receiver node.
[Feature #19070]
Impacts on memory usage and performance are below:
Memory usage:
```
$ cat test.rb
root = RubyVM::AbstractSyntaxTree.parse_file(File.expand_path('../test/ruby/test_keyword.rb', __FILE__), keep_tokens: true)
$ /usr/bin/time -f %Mkb /usr/local/bin/ruby -v
ruby 3.2.0dev (2022-11-19T09:41:54Z 19070-keep_tokens d3af1b8057) [x86_64-linux]
11408kb
# keep_tokens :false
$ /usr/bin/time -f %Mkb /usr/local/bin/ruby test.rb
17508kb
# keep_tokens :true
$ /usr/bin/time -f %Mkb /usr/local/bin/ruby test.rb
30960kb
```
Performance:
```
$ cat ../ast_keep_tokens.yml
prelude: |
src = <<~SRC
module M
class C
def m1(a, b)
1 + a + b
end
end
end
SRC
benchmark:
without_keep_tokens: |
RubyVM::AbstractSyntaxTree.parse(src, keep_tokens: false)
with_keep_tokens: |
RubyVM::AbstractSyntaxTree.parse(src, keep_tokens: true)
$ make benchmark COMPARE_RUBY="./ruby" ARGS=../ast_keep_tokens.yml
/home/kaneko.y/.rbenv/shims/ruby --disable=gems -rrubygems -I../benchmark/lib ../benchmark/benchmark-driver/exe/benchmark-driver \
--executables="compare-ruby::./ruby -I.ext/common --disable-gem" \
--executables="built-ruby::./miniruby -I../lib -I. -I.ext/common ../tool/runruby.rb --extout=.ext -- --disable-gems --disable-gem" \
--output=markdown --output-compare -v ../ast_keep_tokens.yml
compare-ruby: ruby 3.2.0dev (2022-11-19T09:41:54Z 19070-keep_tokens d3af1b8057) [x86_64-linux]
built-ruby: ruby 3.2.0dev (2022-11-19T09:41:54Z 19070-keep_tokens d3af1b8057) [x86_64-linux]
warming up..
| |compare-ruby|built-ruby|
|:--------------------|-----------:|---------:|
|without_keep_tokens | 21.659k| 21.303k|
| | 1.02x| -|
|with_keep_tokens | 6.220k| 5.691k|
| | 1.09x| -|
```
* Folding results should not be empty.
If `OnigCodePointCount(to->n)` were 0, `for` loop using `fn`
wouldn't execute and `ncs` elements are not initialized.
```
enc/unicode.c:557:21: warning: 'ncs[0]' may be used uninitialized in this function [-Wmaybe-uninitialized]
557 | for (i = 0; i < ncs[0]; i++) {
| ~~~^~~
```
* Cast to `enum yytokentype`
Additional enums for scanner events by ripper are not included
in `yytokentype`.
```
ripper.y:7274:28: warning: implicit conversion from 'enum <anonymous>' to 'enum yytokentype' [-Wenum-conversion]
```
Tabs were expanded because previously the file did not have any tab indentation.
Please update your editor config, and use misc/expand_tabs.rb in the pre-commit hook.
* include/ruby/ruby.h (UNREACHABLE_RETURN): UNREACHABLE at the end
of non-void functions.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64025 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* parse.y: use tSP same as ripper instead of tSPACE.
[ruby-core:86080] [Bug #14597]
* ext/ripper/eventids2.c: tSP is defined in ripper.c now.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62727 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* defs/id.def, parse.y: Switch internal token name to reflect
current form of safe-call operator. [Fix GH-1090]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@52525 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* ext/ripper/eventids2.c (token_assoc): reduce size. unsigned
short can hold 65535 at least.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48169 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* parse.y (assoc): allow quoted ID as a key of a hash literal.
[ruby-core:34453] [Feature #4276]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@47649 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* ext/ripper/eventids2.c (ripper_init_eventids2): ripper support for
new literals, tRATIONAL and tIMAGINARY.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@42330 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
support. %i{ .. } returns a list of symbols without interpolation,
%I{ .. } returns a list of symbols with interpolation. Thanks to
Josh Susser for inspiration of this feature. [Feature #4985]
* ext/ripper/eventids2.c: added ripper events for %i and %I.
* test/ripper/test_parser_events.rb: ripper tests
* test/ripper/test_scanner_events.rb: ditto
* test/ruby/test_array.rb: test for %i and %I behavior
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@36524 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
string (e.g. ?x).
* lib/tempfile.rb (Tempfile::make_tmpname): put dot between
basename and pid. [ruby-talk:196272]
* parse.y (do_block): remove -> style block.
* parse.y (parser_yylex): remove tLAMBDA_ARG.
* eval.c (rb_call0): binding for the return event hook should have
consistent scope. [ruby-core:07928]
* eval.c (proc_invoke): return behavior should depend whether it
is surrounded by a lambda or a mere block.
* eval.c (formal_assign): handles post splat arguments.
* eval.c (rb_call0): ditto.
* st.c (strhash): use FNV-1a hash.
* parse.y (parser_yylex): removed experimental ';;' terminator.
* eval.c (rb_node_arity): should be aware of post splat arguments.
* eval.c (rb_proc_arity): ditto.
* parse.y (f_args): syntax rule enhanced to support arguments
after the splat.
* parse.y (block_param): ditto for block parameters.
* parse.y (f_post_arg): mandatory formal arguments after the splat
argument.
* parse.y (new_args_gen): generate nodes for mandatory formal
arguments after the splat argument.
* eval.c (rb_eval): dispatch mandatory formal arguments after the
splat argument.
* parse.y (args): allow more than one splat in the argument list.
* parse.y (method_call): allow aref [] to accept all kind of
method argument, including assocs, splat, and block argument.
* eval.c (SETUP_ARGS0): prepare block argument as well.
* lib/mathn.rb (Integer): remove Integer#gcd2. [ruby-core:07931]
* eval.c (error_line): print receivers true/false/nil specially.
* eval.c (rb_proc_yield): handles parameters in yield semantics.
* eval.c (nil_yield): gives LocalJumpError to denote no block
error.
* io.c (rb_io_getc): now takes one-character string.
* string.c (rb_str_hash): use FNV-1a hash from Fowler/Noll/Vo
hashing algorithm.
* string.c (rb_str_aref): str[0] now returns 1 character string,
instead of a fixnum. [Ruby2]
* parse.y (parser_yylex): ?c now returns 1 character string,
instead of a fixnum. [Ruby2]
* string.c (rb_str_aset): no longer support fixnum insertion.
* eval.c (umethod_bind): should not update original class.
[ruby-dev:28636]
* eval.c (ev_const_get): should support constant access from
within instance_eval(). [ruby-dev:28327]
* time.c (time_timeval): should round for usec floating
number. [ruby-core:07896]
* time.c (time_add): ditto.
* dir.c (sys_warning): should not call a vararg function
rb_sys_warning() indirectly. [ruby-core:07886]
* numeric.c (flo_divmod): the first element of Float#divmod should
be an integer. [ruby-dev:28589]
* test/ruby/test_float.rb: add tests for divmod, div, modulo and remainder.
* re.c (rb_reg_initialize): should not allow modifying literal
regexps. frozen check moved from rb_reg_initialize_m as well.
* re.c (rb_reg_initialize): should not modify untainted objects in
safe levels higher than 3.
* re.c (rb_memcmp): type change from char* to const void*.
* dir.c (dir_close): should not close untainted dir stream.
* dir.c (GetDIR): add tainted/frozen check for each dir operation.
* lib/rdoc/parsers/parse_rb.rb (RDoc::RubyParser::parse_symbol_arg):
typo fixed. a patch from Florian Gross <florg at florg.net>.
* eval.c (EXEC_EVENT_HOOK): trace_func may remove itself from
event_hooks. no guarantee for arbitrary hook deletion.
[ruby-dev:28632]
* util.c (ruby_strtod): differ addition to minimize error.
[ruby-dev:28619]
* util.c (ruby_strtod): should not raise ERANGE when the input
string does not have any digits. [ruby-dev:28629]
* eval.c (proc_invoke): should restore old ruby_frame->block.
thanks to ts <decoux at moulon.inra.fr>. [ruby-core:07833]
also fix [ruby-dev:28614] as well.
* signal.c (trap): sig should be less then NSIG. Coverity found
this bug. a patch from Kevin Tew <tewk at tewk.com>.
[ruby-core:07823]
* math.c (math_log2): add new method inspired by
[ruby-talk:191237].
* math.c (math_log): add optional base argument to Math::log().
[ruby-talk:191308]
* ext/syck/emitter.c (syck_scan_scalar): avoid accessing
uninitialized array element. a patch from Pat Eyler
<rubypate at gmail.com>. [ruby-core:07809]
* array.c (rb_ary_fill): initialize local variables first. a
patch from Pat Eyler <rubypate at gmail.com>. [ruby-core:07810]
* ext/syck/yaml2byte.c (syck_yaml2byte_handler): need to free
type_tag. a patch from Pat Eyler <rubypate at gmail.com>.
[ruby-core:07808]
* ext/socket/socket.c (make_hostent_internal): accept ai_family
check from Sam Roberts <sroberts at uniserve.com>.
[ruby-core:07691]
* util.c (ruby_strtod): should not cut off 18 digits for no
reason. [ruby-core:07796]
* array.c (rb_ary_fill): internalize local variable "beg" to
pacify Coverity. [ruby-core:07770]
* pack.c (pack_unpack): now supports CRLF newlines. a patch from
<tommy at tmtm.org>. [ruby-dev:28601]
* applied code clean-up patch from Stefan Huehner
<stefan at huehner.org>. [ruby-core:07764]
* lib/jcode.rb (String::tr_s): should have translated non
squeezing character sequence (i.e. a character) as well. thanks
to Hiroshi Ichikawa <gimite at gimite.ddo.jp> [ruby-list:42090]
* ext/socket/socket.c: document update patch from Sam Roberts
<sroberts at uniserve.com>. [ruby-core:07701]
* lib/mathn.rb (Integer): need not to remove gcd2. a patch from
NARUSE, Yui <naruse at airemix.com>. [ruby-dev:28570]
* parse.y (arg): too much NEW_LIST()
* eval.c (SETUP_ARGS0): remove unnecessary access to nd_alen.
* eval.c (rb_eval): use ARGSCAT for NODE_OP_ASGN1.
[ruby-dev:28585]
* parse.y (arg): use NODE_ARGSCAT for placeholder.
* lib/getoptlong.rb (GetoptLong::get): RDoc update patch from
mathew <meta at pobox.com>. [ruby-core:07738]
* variable.c (rb_const_set): raise error when no target klass is
supplied. [ruby-dev:28582]
* prec.c (prec_prec_f): documentation patch from
<gerardo.santana at gmail.com>. [ruby-core:07689]
* bignum.c (rb_big_pow): second operand may be too big even if
it's a Fixnum. [ruby-talk:187984]
* README.EXT: update symbol description. [ruby-talk:188104]
* COPYING: explicitly note GPLv2. [ruby-talk:187922]
* parse.y: remove some obsolete syntax rules (unparenthesized
method calls in argument list).
* eval.c (rb_call0): insecure calling should be checked for non
NODE_SCOPE method invocations too.
* eval.c (rb_alias): should preserve the current safe level as
well as method definition.
* process.c (rb_f_sleep): remove RDoc description about SIGALRM
which is not valid on the current implementation. [ruby-dev:28464]
Thu Mar 23 21:40:47 2006 K.Kosako <sndgk393 AT ybb.ne.jp>
* eval.c (method_missing): should support argument splat in
super. a bug in combination of super, splat and
method_missing. [ruby-talk:185438]
* configure.in: Solaris SunPro compiler -rapth patch from
<kuwa at labs.fujitsu.com>. [ruby-dev:28443]
* configure.in: remove enable_rpath=no for Solaris.
[ruby-dev:28440]
* ext/win32ole/win32ole.c (ole_val2olevariantdata): change behavior
of converting OLE Variant object with VT_ARRAY|VT_UI1 and Ruby
String object.
* ruby.1: a clarification patch from David Lutterkort
<dlutter at redhat.com>. [ruby-core:7508]
* lib/rdoc/ri/ri_paths.rb (RI::Paths): adding paths from rubygems
directories. a patch from Eric Hodel <drbrain at segment7.net>.
[ruby-core:07423]
* eval.c (rb_clear_cache_by_class): clearing wrong cache.
* ext/extmk.rb: use :remove_destination to install extension libraries
to avoid SEGV. [ruby-dev:28417]
* eval.c (rb_thread_fd_writable): should not re-schedule output
from KILLED thread (must be error printing).
* array.c (rb_ary_flatten_bang): allow specifying recursion
level. [ruby-talk:182170]
* array.c (rb_ary_flatten): ditto.
* gc.c (add_heap): a heap_slots may overflow. a patch from Stefan
Weil <weil at mail.berlios.de>.
* eval.c (rb_call): use separate cache for fcall/vcall
invocation.
* eval.c (rb_eval): NODE_FCALL, NODE_VCALL can call local
functions.
* eval.c (rb_mod_local): a new method to specify newly added
visibility "local".
* eval.c (search_method): search for local methods which are
visible only from the current class.
* class.c (rb_class_local_methods): a method to list local methods.
* object.c (Init_Object): add BasicObject class as a top level
BlankSlate class.
* ruby.h (SYM2ID): should not cast to signed long.
[ruby-core:07414]
* class.c (rb_include_module): allow module duplication.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@10235 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* ext/ripper/MANIFEST: new file.
* ext/ripper/README: new file.
* ext/ripper/depend: new file.
* ext/ripper/extconf.rb: new file.
* ext/ripper/eventids2.c: new file.
* ext/ripper/ripper.rb.in: new file.
* ext/ripper/lib/ripper.rb: new file.
* ext/ripper/test/check-event-arity.rb: new file.
* ext/ripper/test/check-event-coverage.sh: new file.
* ext/ripper/test/check-scanner-event-coverage.rb: new file.
* ext/ripper/test/list-called-events.rb: new file.
* ext/ripper/test/src_rb: new file.
* ext/ripper/test/validate.rb: new file.
* ext/ripper/tools/generate-eventids1.rb: new file.
* ext/ripper/tools/generate-param-macros.rb: new file.
* ext/ripper/tools/generate-ripper_rb.rb: new file.
* ext/ripper/tools/list-parse-event-ids.rb: new file.
* ext/ripper/tools/list-scan-event-ids.rb: new file.
* ext/ripper/tools/preproc.rb: new file.
* ext/ripper/tools/strip.rb: new file.
* test/ripper: ripper tests added.
* test/ripper/dummyparser.rb: new file.
* test/ripper/test_parser_events.rb: new file.
* test/ripper/test_scanner_events.rb: new file.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@6891 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Nobuyoshi Nakada
yui-knk
git
nobu
charliesome
tenderlove
drbrain
matz
aamine