Граф коммитов

252 Коммитов

Автор SHA1 Сообщение Дата
yui-knk 8be62f06c8 Remove ruby2_keywords related to args forwarding
This was introduced by b609bdeb53
to suppress warnings. However these warngins were deleted by
beae6cbf0f. Therefore these codes
are not needed anymore.
2022-11-29 15:39:56 +09:00
yui-knk d8601621ed Enhance keep_tokens option for RubyVM::AbstractSyntaxTree parsing methods
Implementation for Language Server Protocol (LSP) sometimes needs token information.
For example both `m(1)` and `m(1, )` has same AST structure other than node locations
then it's impossible to check the existence of `,` from AST. However in later case,
it might be better to suggest variables list for the second argument.
Token information is important for such case.

This commit adds these methods.

* Add `keep_tokens` option for `RubyVM::AbstractSyntaxTree.parse`, `.parse_file` and `.of`
* Add `RubyVM::AbstractSyntaxTree::Node#tokens` which returns tokens for the node including tokens for descendants nodes.
* Add `RubyVM::AbstractSyntaxTree::Node#all_tokens` which returns all tokens for the input script regardless the receiver node.

[Feature #19070]

Impacts on memory usage and performance are below:

Memory usage:

```
$ cat test.rb
root = RubyVM::AbstractSyntaxTree.parse_file(File.expand_path('../test/ruby/test_keyword.rb', __FILE__), keep_tokens: true)

$ /usr/bin/time -f %Mkb /usr/local/bin/ruby -v
ruby 3.2.0dev (2022-11-19T09:41:54Z 19070-keep_tokens d3af1b8057) [x86_64-linux]
11408kb

# keep_tokens :false
$ /usr/bin/time -f %Mkb /usr/local/bin/ruby test.rb
17508kb

# keep_tokens :true
$ /usr/bin/time -f %Mkb /usr/local/bin/ruby test.rb
30960kb
```

Performance:

```
$ cat ../ast_keep_tokens.yml
prelude: |
  src = <<~SRC
    module M
      class C
        def m1(a, b)
          1 + a + b
        end
      end
    end
  SRC
benchmark:
  without_keep_tokens: |
    RubyVM::AbstractSyntaxTree.parse(src, keep_tokens: false)
  with_keep_tokens: |
    RubyVM::AbstractSyntaxTree.parse(src, keep_tokens: true)

$ make benchmark COMPARE_RUBY="./ruby" ARGS=../ast_keep_tokens.yml
/home/kaneko.y/.rbenv/shims/ruby --disable=gems -rrubygems -I../benchmark/lib ../benchmark/benchmark-driver/exe/benchmark-driver \
            --executables="compare-ruby::./ruby -I.ext/common --disable-gem" \
            --executables="built-ruby::./miniruby -I../lib -I. -I.ext/common  ../tool/runruby.rb --extout=.ext  -- --disable-gems --disable-gem" \
            --output=markdown --output-compare -v ../ast_keep_tokens.yml
compare-ruby: ruby 3.2.0dev (2022-11-19T09:41:54Z 19070-keep_tokens d3af1b8057) [x86_64-linux]
built-ruby: ruby 3.2.0dev (2022-11-19T09:41:54Z 19070-keep_tokens d3af1b8057) [x86_64-linux]
warming up..

|                     |compare-ruby|built-ruby|
|:--------------------|-----------:|---------:|
|without_keep_tokens  |     21.659k|   21.303k|
|                     |       1.02x|         -|
|with_keep_tokens     |      6.220k|    5.691k|
|                     |       1.09x|         -|
```
2022-11-21 09:01:34 +09:00
yui-knk f0ce118662 Make anonymous rest arg (*) and block arg (&) accessible from ARGS node 2022-11-18 18:25:42 +09:00
yui-knk 561b6c4fa0 Remove unused macro
NEW_PREEXE has not been used since 52a5f76e8b
2022-10-09 14:00:52 +09:00
yui-knk 4bfdf6d06d Move `error` from top_stmts and top_stmt to stmt
By this change, syntax error is recovered smaller units.
In the case below, "DEFN :bar" is same level with "CLASS :Foo"
now.

```
module Z
  class Foo
    foo.
  end

  def bar
  end
end
```

[Feature #19013]
2022-10-08 17:59:11 +09:00
Takashi Kokubun 5b21e94beb Expand tabs [ci skip]
[Misc #18891]
2022-07-21 09:42:04 -07:00
Nobuyoshi Nakada 54f0e63a8c Remove `NODE_DASGN_CURR` [Feature #18406]
This `NODE` type was used in pre-YARV implementation, to improve
the performance of assignment to dynamic local variable defined at
the innermost scope.  It has no longer any actual difference with
`NODE_DASGN`, except for the node dump.
2021-12-13 12:53:03 +09:00
Nobuyoshi Nakada d118e7c025
Turn nd_type_p into an inline function 2021-12-04 10:35:44 +09:00
S.H ec7f14d9fa
Add `nd_type_p` macro 2021-12-04 00:01:24 +09:00
Yusuke Endoh feda058531 Refactor hacky ID tables to struct rb_ast_id_table_t
The implementation of a local variable tables was represented as `ID*`,
but it was very hacky: the first element is not an ID but the size of
the table, and, the last element is (sometimes) a link to the next local
table only when the id tables are a linked list.

This change converts the hacky implementation to a normal struct.
2021-11-21 08:59:24 +09:00
Yusuke Endoh 6764256dc7 node/h: clean node field accessors
This change removes nd_oid, nd_rest, and nd_opt, and adds some comments
for special accessors.
2021-11-17 23:39:34 +09:00
Yusuke Endoh fb01411ae8 node.h: Reduce struct size to fit with Ruby object size (five VALUEs)
by merging `rb_ast_body_t#line_count` and `#script_lines`.

Fortunately `line_count == RARRAY_LEN(script_lines)` was always
satisfied. When script_lines is saved, it has an array of lines, and
when not saved, it has a Fixnum that represents the old line_count.
2021-06-18 02:34:27 +09:00
Yusuke Endoh acae5f363d ast.rb: RubyVM::AST.parse and .of accepts `save_script_lines: true`
This option makes the parser keep the original source as an array of
the original code lines. This feature exploits the mechanism of
`SCRIPT_LINES__` but records only the specified code that is passed to
RubyVM::AST.of or .parse, instead of recording all parsed program texts.
2021-06-18 02:34:27 +09:00
Nobuyoshi Nakada c060bdc2b4
NODE markability should not change by nd_set_type 2021-01-14 16:12:02 +09:00
Kazuki Tsujimoto e03e1982bd
Change NODE layout for pattern matching
I prefer pconst to be the first element of NODE.

  Before:

       | ARYPTN | FNDPTN | HSHPTN
    ---+--------+--------+-----------
    u1 | imemo  | imemo  | pkwargs
    u2 | pconst | pconst | pconst
    u3 | apinfo | fpinfo | pkwrestarg

  After:

       | ARYPTN | FNDPTN | HSHPTN
    ---+--------+--------+-----------
    u1 | pconst | pconst | pconst
    u2 | imemo  | imemo  | pkwargs
    u3 | apinfo | fpinfo | pkwrestarg
2020-11-01 16:19:07 +09:00
卜部昌平 cd1d6d9029 include/ruby/backward/2/r_cast.h: deprecate
Remove all usages of RCAST() so that the header file can be excluded
from ruby/ruby.h's dependency.
2020-08-27 15:03:36 +09:00
Aaron Patterson e8edc34f0a Remove unused struct member
I accidentally added this in 35ba2783fe,
and it's making the size of RVALUE be too big. I'm sorry! orz
2020-08-03 17:01:13 -07:00
Kazuki Tsujimoto fcdbdff631
rb_{ary,fnd}_pattern_info: Remove imemo member to reduce memory usage
This is a partial revert commit of 8f096226e1.

NODE layout:

  Before:

       | ARYPTN | FNDPTN | HSHPTN
    ---+--------+--------+-----------
    u1 | pconst | pconst | pconst
    u2 | unused | unused | pkwargs
    u3 | apinfo | fpinfo | pkwrestarg

  After:

       | ARYPTN | FNDPTN | HSHPTN
    ---+--------+--------+-----------
    u1 | imemo  | imemo  | pkwargs
    u2 | pconst | pconst | pconst
    u3 | apinfo | fpinfo | pkwrestarg
2020-08-02 01:04:06 +09:00
Aaron Patterson 35ba2783fe Use a linked list to eliminate imemo tmp bufs for managing local tables
This patch changes local table memory to be managed by a linked list
rather than via the garbage collector.  It reduces allocations from the
GC and also fixes a use-after-free bug in the concurrent-with-sweep
compactor I'm working on.
2020-07-27 12:40:01 -07:00
Koichi Sasada a0f12a0258
Use ID instead of GENTRY for gvars. (#3278)
Use ID instead of GENTRY for gvars.

Global variables are compiled into GENTRY (a pointer to struct
rb_global_entry). This patch replace this GENTRY to ID and
make the code simple.

We need to search GENTRY from ID every time (st_lookup), so
additional overhead will be introduced.
However, the performance of accessing global variables is not
important now a day and this simplicity helps Ractor development.
2020-07-03 16:56:44 +09:00
Kazuki Tsujimoto ddded1157a
Introduce find pattern [Feature #16828] 2020-06-14 09:24:36 +09:00
Nobuyoshi Nakada 0a52015da7
Constified code_loc_gen 2020-05-14 17:15:24 +09:00
Kazuki Tsujimoto 6ed7bc83ec
Fix indentation 2020-05-04 13:27:25 +09:00
卜部昌平 4ff3f20540 add #include guard hack
According to MSVC manual (*1), cl.exe can skip including a header file
when that:

- contains #pragma once, or
- starts with #ifndef, or
- starts with #if ! defined.

GCC has a similar trick (*2), but it acts more stricter (e. g. there
must be _no tokens_ outside of #ifndef...#endif).

Sun C lacked #pragma once for a looong time.  Oracle Developer Studio
12.5 finally implemented it, but we cannot assume such recent version.

This changeset modifies header files so that each of them include
strictly one #ifndef...#endif.  I believe this is the most portable way
to trigger compiler optimizations. [Bug #16770]

*1: https://docs.microsoft.com/en-us/cpp/preprocessor/once
*2: https://gcc.gnu.org/onlinedocs/cppinternals/Guard-Macros.html
2020-04-13 16:06:00 +09:00
Jeremy Evans ffd0820ab3 Deprecate taint/trust and related methods, and make the methods no-ops
This removes the related tests, and puts the related specs behind
version guards.  This affects all code in lib, including some
libraries that may want to support older versions of Ruby.
2019-11-18 01:00:25 +02:00
卜部昌平 c9ffe751d1 delete unused functions
Looking at the list of symbols inside of libruby-static.a, I found
hundreds of functions that are defined, but used from nowhere.

There can be reasons for each of them (e.g. some functions are
specific to some platform, some are useful when debugging, etc).
However it seems the functions deleted here exist for no reason.

This changeset reduces the size of ruby binary from 26,671,456
bytes to 26,592,864 bytes on my machine.
2019-11-14 20:35:48 +09:00
Nobuyoshi Nakada fb6a489af2
Revert "Method reference operator"
This reverts commit 67c5747369.
[Feature #16275]
2019-11-12 17:24:48 +09:00
Nobuyoshi Nakada b609bdeb53
Define arguments forwarding as `ruby2_keywords` style
Get rid of these redundant and useless warnings.

```
$ ruby -e 'def bar(a) a; end; def foo(...) bar(...) end; foo({})'
-e:1: warning: The last argument is used as the keyword parameter
-e:1: warning: for `foo' defined here
-e:1: warning: The keyword argument is passed as the last hash parameter
-e:1: warning: for `bar' defined here
```
2019-10-25 01:16:05 +09:00
Aaron Patterson 293c6c8cc3
Add compaction support to `rb_ast_t`
This commit adds compaction support to `rb_ast_t`.
2019-09-26 15:41:46 -07:00
Nobuyoshi Nakada e81a3e6df5
Allows calling a private method only with bare `self` 2019-09-20 22:05:54 +09:00
Nobuyoshi Nakada 5f5aca1b5f
Moved ruby_node_name declaration to node.h 2019-09-13 17:20:11 +09:00
Aaron Patterson 91ee9584f9
Macros can't be expressions, so make a function
Macros can't be expressions, that is a GNU extension (I didn't know
that).  This commit converts the macro to a function so that everything
will compile correctly on non-GNU compatible compilers.
2019-09-10 14:00:48 -07:00
Aaron Patterson 139510238b
WB needs to be executed after object is reachable 2019-09-10 12:42:35 -07:00
Aaron Patterson 4524780d17
Revert "Reverting node marking until I can fix GC problem."
This reverts commit 092f31e7e2.
2019-09-09 14:26:51 -07:00
Yusuke Endoh 99c9431ea1 Rename NODE_ARRAY to NODE_LIST to reflect its actual use cases
and NODE_ZARRAY to NODE_ZLIST.

NODE_ARRAY is used not only by an Array literal, but also the contents
of Hash literals, method call arguments, dynamic string literals, etc.
In addition, the structure of NODE_ARRAY is a linked list, not an array.

This is very confusing, so I believe `NODE_LIST` is a better name.
2019-09-07 13:56:29 +09:00
Aaron Patterson 092f31e7e2
Reverting node marking until I can fix GC problem.
Looks like we're getting WB misses during stressful GC on startup.  I am
investigating.
2019-09-05 12:44:23 -07:00
Aaron Patterson 8f096226e1
Stash tmpbuffer inside internal structs
I guess those AST node were actually used for something, so we'd better
not touch them.  Instead this commit just puts the tmpbuffer inside a
different internal struct so that we can mark them.
2019-09-05 11:04:43 -07:00
Aaron Patterson 545b6db3fb
Create two buckets for allocating NODE structs
This commit adds two buckets for allocating NODE structs, then allocates
"markable" NODE objects from one bucket.  The reason to do this is so
when the AST mark function scans nodes for VALUE objects to mark, we
only scan NODE objects that we know to reference VALUE objects.  If we
*did not* divide the objects, then the mark function spends too much
time scanning objects that don't contain any references.
2019-09-05 10:13:50 -07:00
Kazuki Tsujimoto 94d6ec1d90
Make pattern matching support **nil syntax 2019-09-01 16:39:34 +09:00
Jeremy Evans 6a9ce1fea8 Support **nil syntax for specifying a method does not accept keyword arguments
This syntax means the method should be treated as a method that
uses keyword arguments, but no specific keyword arguments are
supported, and therefore calling the method with keyword arguments
will raise an ArgumentError.  It is still allowed to double splat
an empty hash when calling the method, as that does not pass
any keyword arguments.
2019-08-30 12:39:31 -07:00
Lourens Naudé 90c4bd2d2b
Let memory sizes of the various IMEMO object types be reflected correctly
[Feature #15805]

Closes: https://github.com/ruby/ruby/pull/2140
2019-07-23 16:22:34 +09:00
Nobuyoshi Nakada a036a8a038
Adjust styles and indents 2019-07-19 06:35:15 +09:00
Martin Dürst f258137083 Fix grammar of macro name: ECCESSED -> ECCESSIVE
Fix the name of the macro variable introduced in 0872ea5330
from NODE_SPECIAL_EXCESSED_COMMA to NODE_SPECIAL_EXCESSIVE_COMMA.
2019-06-05 14:03:50 +09:00
Yusuke Endoh 0872ea5330 node.h: Avoid a magic number to represent excessed comma
`(ID)1` was assigned to NODE_ARGS#rest_arg for `{|x,| }`.
This change removes the magic number by introducing an explicit macro
variable for it: NODE_SPECIAL_EXCESSED_COMMA.
2019-06-04 23:17:18 +09:00
Nobuyoshi Nakada ff21e75d32
parse.y: duplicated when clause warning
* parse.y (case_args): moved "duplicated when clause" warning from
  compile phase, so that `ruby -wc` shows them.
2019-05-05 00:29:12 +09:00
ktsj 9738f96fcf Introduce pattern matching [EXPERIMENTAL]
[ruby-core:87945] [Feature #14912]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67586 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-04-17 06:48:03 +00:00
mame c36a6f97f2 node.h: introduce nd_brace to determine if a hash literal is a keyword
NODE_HASH#nd_brace is a flag that is 1 for `foo({ k: 1 })` and 0 for
`foo(k: 1)`.
nd_alen had been abused for the flag (and the implementation is
completely the same), but an explicit name is better to read.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67266 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-03-15 05:19:54 +00:00
nobu 67c5747369 Method reference operator
Introduce the new operator for method reference, `.:`.
[Feature #12125] [Feature #13581]
[EXPERIMENTAL]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66667 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-12-31 15:00:37 +00:00
nobu 6e610f5ea7 Parse the source in SCRIPT_LINES__ as array
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65653 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-11-10 11:40:33 +00:00
yui-knk ce798d08de Implement `RubyVM::AST.of` [Feature #14836]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65542 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-11-05 02:13:45 +00:00