github
/
ruby
зеркало из https://github.com/github/ruby.git
1
0
Форкнуть 0
Код Задачи Пакеты Проекты Релизы Вики Активность
85 800 коммитов 219 Ветки 1 255 Теги 478 MiB
Граф коммитов

129 Коммитов

Автор SHA1 Сообщение Дата
S-H-GAMELINKS 0cc5f77c07 Remove unused functions from struct rb_parser_config_struct 
StringValueCStr has not used in parse.y
 2024-06-02 09:50:43 +09:00
Nobuyoshi Nakada cedc7737b6 Make interchangeable NODE types aliases 2024-06-02 09:43:33 +09:00
Yusuke Endoh a15e4d405b Revert 528c4501f4 
Recently, `TestRubyLiteral#test_float` fails randomly.

```
  1) Error:
TestRubyLiteral#test_float:
ArgumentError: SyntaxError#path changed: "(eval at /home/chkbuild/chkbuild/tmp/build/20240527T050036Z/ruby/test/ruby/test_literal.rb:642)"->"(eval at /home/chkbuild/chkbuild/tmp/build/20240527T050036Z/ruby/test/ruby/test_literal.rb:642)"
```
https://rubyci.s3.amazonaws.com/s390x/ruby-master/log/20240527T050036Z.fail.html.gz

According to Launchable, the first failure was on Apr 30.
This is just when 528c4501f4 was
committed. I don't know if the change is really the cause, but I want to
revert it once to see if the random failure disappears.
 2024-05-31 18:24:43 +09:00
Nobuyoshi Nakada 3c16d93cd3 Constify encoding type in universal parser 
Fixed warning about discarding modifiers.

```
../src/ruby_parser.c:677:48: warning: passing 'rb_encoding *' (aka 'const struct OnigEncodingTypeST *') to parameter of type 'void *' discards qualifiers [-Wincompatible-pointer-types-discards-qualifiers]
  677 |     ast = rb_parser_compile(p, gets, ptr, len, enc, input, line);
      |                                                ^~~
../src/internal/parse.h:58:128: note: passing argument to parameter 'fname_enc' here
   58 | rb_ast_t *rb_parser_compile(rb_parser_t *p, rb_parser_lex_gets_func *gets, const char *fname_ptr, long fname_len, rb_encoding *fname_enc, rb_parser_input_data input, int line);
      |                                                                                                                                ^
```
 2024-05-13 08:26:54 +09:00
yui-knk 528c4501f4 Use `rb_parser_string_t *` as `ruby_sourcefile_string` 
This reduces dependency on VALUE.
 2024-04-30 09:00:05 +09:00
yui-knk 29aaf4abe6 Remove `ast_new` field from `struct rb_parser_config_struct` 
`ast_new` can be embedded into `rb_ast_new`.
 2024-04-28 17:58:57 +09:00
HASUMI Hitoshi ddd8da4b6b [Universal parser] Improve AST structure 
This patch moves `ast->node_buffer->config` to `ast->config` aiming to improve readability and maintainability of the source.

## Background

We could not add the `config` field to the `rb_ast_t *` due to the five-word restriction of the IMEMO object.
But it is now doable by merging https://github.com/ruby/ruby/pull/10618

## About assigning `&rb_global_parser_config` to `ast->config` in `ast_alloc()`

The approach of not setting `ast->config` in `ast_alloc()` means that the client, CRuby in this scenario, that directly calls `ast_alloc()` will be responsible for releasing it if a resource that is passed to AST needs to be released.

However, we have put on hold whether we can guarantee the above so far, thus, this patch looks like that.

```
// ruby_parser.c
static VALUE
ast_alloc(void)
{
    rb_ast_t *ast;
    VALUE vast = TypedData_Make_Struct(0, rb_ast_t, &ast_data_type, ast);
#ifdef UNIVERSAL_PARSER
    ast = (rb_ast_t *)DATA_PTR(vast);
    ast->config = &rb_global_parser_config;
#endif
    return vast;
}
```
 2024-04-28 12:08:21 +09:00
HASUMI Hitoshi 55a402bb75 Add line_count field to rb_ast_body_t 
This patch adds `int line_count` field to `rb_ast_body_t` structure.
Instead, we no longer cast `script_lines` to Fixnum.

## Background

Ref https://github.com/ruby/ruby/pull/10618

In the PR above, we have decoupled IMEMO from `rb_ast_t`.
This means we could lift the five-words-restriction of the structure
that forced us to unionize `rb_ast_t *` and `FIXNUM` in one field.

## Relating refactor

- Remove the second parameter of `rb_ruby_ast_new()` function

## Attention

I will remove a code that assigns -1 to line_count, in `rb_binding_add_dynavars()`
of vm.c, because I don't think it is necessary.
But I will make another PR for this so that we can atomically revert
in case I was wrong (See the comment on the code)
 2024-04-27 12:08:26 +09:00
HASUMI Hitoshi 2244c58b00 [Universal parser] Decouple IMEMO from rb_ast_t 
This patch removes the `VALUE flags` member from the `rb_ast_t` structure making `rb_ast_t` no longer an IMEMO object.

## Background

We are trying to make the Ruby parser generated from parse.y a universal parser that can be used by other implementations such as mruby.
To achieve this, it is necessary to exclude VALUE and IMEMO from parse.y, AST, and NODE.

## Summary (file by file)

- `rubyparser.h`
  - Remove the `VALUE flags` member from `rb_ast_t`
- `ruby_parser.c` and `internal/ruby_parser.h`
  - Use TypedData_Make_Struct VALUE which wraps `rb_ast_t` `in ast_alloc()` so that GC can manage it
    - You can retrieve `rb_ast_t` from the VALUE by `rb_ruby_ast_data_get()`
  - Change the return type of `rb_parser_compile_XXXX()` functions from `rb_ast_t *` to `VALUE`
  - rb_ruby_ast_new() which internally `calls ast_alloc()` is to create VALUE vast outside ruby_parser.c
- `iseq.c` and `vm_core.h`
  - Amend the first parameter of `rb_iseq_new_XXXX()` functions from `rb_ast_body_t *` to `VALUE`
  - This keeps the VALUE of AST on the machine stack to prevent being removed by GC
- `ast.c`
  - Almost all change is replacement `rb_ast_t *ast` with `VALUE vast` (sorry for the big diff)
  - Fix `node_memsize()`
    - Now it includes `rb_ast_local_table_link`, `tokens` and script_lines
- `compile.c`, `load.c`, `node.c`, `parse.y`, `proc.c`, `ruby.c`, `template/prelude.c.tmpl`, `vm.c` and `vm_eval.c`
  - Follow-up due to the above changes
- `imemo.{c|h}`
  - If an object with `imemo_ast` appears, considers it a bug

Co-authored-by: Nobuyoshi Nakada <nobu@ruby-lang.org>
 2024-04-26 11:21:08 +09:00
yui-knk 813e12557a Remove unused functions from `struct rb_parser_config_struct` 2024-04-24 07:59:51 +09:00
yui-knk e61f76ce9e Remove unused functions from `struct rb_parser_config_struct` 2024-04-23 13:11:46 +09:00
yui-knk 3aa046803d Remove unused functions from `struct rb_parser_config_struct` 2024-04-23 11:18:52 +09:00
yui-knk c7d9376cb5 Remove unused functions from `struct rb_parser_config_struct` 2024-04-23 07:20:22 +09:00
yui-knk 2992e1074a Refactor parser compile functions 
Refactor parser compile functions to reduce the dependence
on ruby functions.
This commit includes these changes

1. Refactor `gets`, `input` and `gets_` of `parser_params`

Parser needs two different data structure to get next line, function (`gets`) and input data (`input`).
However `gets_` is used for both function (`call`) and input data (`ptr`).
`call` is used for managing general callback function when `rb_ruby_parser_compile_generic` is used.
`ptr` is used for managing the current pointer on String when `parser_compile_string` is used.
This commit changes parser to used only `gets` and `input` then removes `gets_`.

2. Move parser_compile functions and `gets` functions from parse.y to ruby_parser.c

This change reduces the dependence on ruby functions from parser.

3. Change ruby_parser and ripper to take care of `VALUE input` GC mark

Move the responsibility of calling `rb_gc_mark` for `VALUE input` from parser to ruby_parser and ripper.
`input` is arbitrary data pointer from the viewpoint of parser.

4. Introduce rb_parser_compile_array function

Caller of `rb_parser_compile_generic` needs to take care about GC because ruby_parser doesn’t know
about the detail of `lex_gets` and `input`.
Introduce `rb_parser_compile_array` to reduce the complexity of ast.c.
 2024-04-23 07:20:22 +09:00
S-H-GAMELINKS 381ce130ae Remove unused functions from struct `rb_parser_config_struct` 2024-04-21 09:55:17 +09:00
yui-knk cee985ea83 Remove unused functions from `struct rb_parser_config_struct` 2024-04-20 18:08:33 +09:00
yui-knk ebaa87f626 Remove unused functions from `struct rb_parser_config_struct` 2024-04-16 07:15:45 +09:00
HASUMI Hitoshi 9b1e97b211 [Universal parser] DeVALUE of p->debug_lines and ast->body.script_lines 
This patch is part of universal parser work.

## Summary
- Decouple VALUE from members below:
  - `(struct parser_params *)->debug_lines`
  - `(rb_ast_t *)->body.script_lines`
- Instead, they are now `rb_parser_ary_t *`
  - They can also be a `(VALUE)FIXNUM` as before to hold line count
- `ISEQ_BODY(iseq)->variable.script_lines` remains VALUE
  - In order to do this,
  - Add `VALUE script_lines` param to `rb_iseq_new_with_opt()`
  - Introduce `rb_parser_build_script_lines_from()` to convert `rb_parser_ary_t *` into `VALUE`

## Other details
- Extend `rb_parser_ary_t *`. It previously could only store `rb_parser_ast_token *`, now can store script_lines, too
- Change tactics of building the top-level `SCRIPT_LINES__` in `yycompile0()`
  - Before: While parsing, each line of the script is added to `SCRIPT_LINES__[path]`
  - After: After `yyparse(p)`, `SCRIPT_LINES__[path]` will be built from `p->debug_lines`
- Remove the second parameter of `rb_parser_set_script_lines()` to make it simple
- Introduce `script_lines_free()` to be called from `rb_ast_free()` because the GC no longer takes care of the script_lines
- Introduce `rb_parser_string_deep_copy()` in parse.y to maintain script_lines when `rb_ruby_parser_free()` called
  - With regard to this, please see *Future tasks* below

## Future tasks
- Decouple IMEMO from `rb_ast_t *`
  - This lifts the five-members-restriction of Ruby object,
  - So we will be able to move the ownership of the `lex.string_buffer` from parser to AST
  - Then we remove `rb_parser_string_deep_copy()` to make the whole thing simple
 2024-04-15 20:51:54 +09:00
yui-knk 924b928e35 Remove unused functions from `struct rb_parser_config_struct` 2024-04-14 09:08:19 +09:00
yui-knk 87ad5ca3ca Remove unused function from `struct rb_parser_config_struct` 2024-04-11 11:56:28 +09:00
yui-knk 4dbd9c7fec Remove unused function from `struct rb_parser_config_struct` 2024-04-09 09:58:05 +09:00
yui-knk 6f7e8e278f Don't set T_TYPES of NODE 
T_TYPES was needed once Ripper jumbled NODEs and other type
objects. However such hack was already removed.
Therefore don't need to set T_TYPES of NODE.
 2024-04-08 11:39:00 +09:00
yui-knk 0b4db711dd Remove unused function from `struct rb_parser_config_struct` 2024-04-06 22:03:26 +09:00
S-H-GAMELINKS dae503874d Remove unused functions from `struct rb_parser_config_struct` 2024-04-06 09:29:44 +09:00
yui-knk 9d1d2de2fe Make `nd_plen` to be int to reduce `rb_long2int` usage 2024-04-05 14:25:04 +09:00
yui-knk bd728a625a Remove unused functions from `struct rb_parser_config_struct` 2024-04-05 10:04:48 +09:00
yui-knk cfd48adb2f Remove unused function from `struct rb_parser_config_struct` 2024-04-04 19:13:03 +09:00
yui-knk f057741c5d NODE_LIT is not used anymore 2024-04-04 13:17:26 +09:00
yui-knk 6056773105 Move shareable_constant_value logic from parse.y to compile.c 2024-04-04 08:44:10 +09:00
yui-knk e816ab0b0c Remove `rb_imemo_tmpbuf_t` from parser 
No parser semantic value types are `VALUE` then no need to
use imemo for managing semantic value stack anymore.
 2024-04-02 19:37:27 +09:00
yui-knk 799e854897 [Feature #20331] Simplify parser warnings for hash keys duplication and when clause duplication 
This commit simplifies warnings for hash keys duplication and when clause duplication,
based on the discussion of https://bugs.ruby-lang.org/issues/20331.
Warnings are reported only when strings are same to ohters.
 2024-04-02 08:26:58 +09:00
S-H-GAMELINKS 0774232bf3 Remove unnecessary macros and functions for Universal Parser 2024-04-01 12:05:16 +09:00
HASUMI Hitoshi 40ecad0ad7
[Universal parser] Fix -Wsuggest-attribute=format warnings 
Under a configuration including `cppflags=-DUNIVERSAL_PARSER`, warnings listed below show in build time:

```
node.c:396:30: warning: initialization left-hand side might be a candidate for a format attribute [-Wsuggest-attribute=format]
  396 |     bug_report_func rb_bug = ast->node_buffer->config->bug;
      |                              ^~~
```

```
ruby_parser.c:655:21: warning: initialization left-hand side might be a candidate for a format attribute [-Wsuggest-attribute=format]
  655 |     .compile_warn = rb_compile_warn,
      |                     ^~~~~~~~~~~~~~~
ruby_parser.c:656:24: warning: initialization left-hand side might be a candidate for a format attribute [-Wsuggest-attribute=format]
  656 |     .compile_warning = rb_compile_warning,
      |                        ^~~~~~~~~~~~~~~~~~
ruby_parser.c:657:12: warning: initialization left-hand side might be a candidate for a format attribute [-Wsuggest-attribute=format]
  657 |     .bug = rb_bug,
      |            ^~~~~~
ruby_parser.c:658:14: warning: initialization left-hand side might be a candidate for a format attribute [-Wsuggest-attribute=format]
  658 |     .fatal = rb_fatal,
      |              ^~~~~~~~
```

To fix, this patch suggests adding `__attribute__((format(printf, n, m)))` to those function declarations.
 2024-03-15 22:14:57 +09:00
HASUMI Hitoshi 9a19cfd4cd [Universal Parser] Reduce dependence on RArray in parse.y 
- Introduce `rb_parser_ary_t` structure to partly eliminate RArray from parse.y
  - In this patch, `parser_params->tokens` and `parser_params->ast->node_buffer->tokens` are now `rb_parser_ary_t *`
  - Instead, `ast_node_all_tokens()` internally creates a Ruby Array object from the `rb_parser_ary_t`
  - Also, delete `rb_ast_tokens()` and `rb_ast_set_tokens()` in node.c

- Implement `rb_parser_str_escape()`
  - This is a port of the `rb_str_escape()` function in string.c
  - `rb_parser_str_escape()` does not depend on `VALUE` (RString)
  - Instead, it uses `rb_parser_stirng_t *`
  - This function works when --dump=y option passed

- Because WIP of the universal parser, similar functions like `rb_parser_tokens_free()` exist in both node.c and parse.y. Refactoring them may be needed in some way in the future

- Although we considered redesigning the structure: `ast->node_buffer->tokens` into `ast->tokens`, we leave it as it is because `rb_ast_t` is an imemo. (We will address it in the future)
 2024-03-12 17:17:52 +09:00
Peter Zhu 01f9b2ae41 Use rb_str_to_interned_str in parse.y 
This commit changes rb_fstring to rb_str_to_interned_str in parse.y.
rb_fstring is private so it shouldn't be used by ripper.
 2024-02-23 13:33:46 -05:00
yui-knk 91cb303531 Remove not used universal parser macros and functions 2024-02-21 13:36:45 +09:00
yui-knk e7ab5d891c Introduce NODE_REGX to manage regexp literal 2024-02-21 08:06:48 +09:00
Peter Zhu c184aa8740 Use rb_gc_mark_and_move for imemo 2024-02-20 10:39:30 -05:00
S-H-GAMELINKS fba647087b Remove uneeded Universal Parser properties 2024-02-20 19:02:24 +09:00
yui-knk 89cfc15207 [Feature #20257] Rearchitect Ripper 
Introduce another semantic value stack for Ripper so that
Ripper can manage both Node and Ruby Object separately.
This rearchitectutre of Ripper solves these issues.
Therefore adding test cases for them.

* [Bug 10436] https://bugs.ruby-lang.org/issues/10436
* [Bug 18988] https://bugs.ruby-lang.org/issues/18988
* [Bug 20055] https://bugs.ruby-lang.org/issues/20055

Checked the differences of `Ripper.sexp` for files under `/test/ruby`
are only on test_pattern_matching.rb.
The differences comes from the differences between
`new_hash_pattern_tail` functions between parser and Ripper.
Ripper `new_hash_pattern_tail` didn’t call `assignable` then
`kw_rest_arg` wasn’t marked as local variable.
This is also fixed by this commit.

```
--- a/./tmp/before/test_pattern_matching.rb
+++ b/./tmp/after/test_pattern_matching.rb
@@ -3607,7 +3607,7 @@
                  [:in,
                   [:hshptn, nil, [], [:var_field, [:@ident, “a”, [984, 13]]]],
                   [[:binary,
-                    [:vcall, [:@ident, “a”, [985, 10]]],
+                    [:var_ref, [:@ident, “a”, [985, 10]]],
                     :==,
                     [:hash, nil]]],
                   nil]]],
@@ -3662,7 +3662,7 @@
                  [:in,
                   [:hshptn, nil, [], [:var_field, [:@ident, “a”, [993, 13]]]],
                   [[:binary,
-                    [:vcall, [:@ident, “a”, [994, 10]]],
+                    [:var_ref, [:@ident, “a”, [994, 10]]],
                     :==,
                     [:hash,
                      [:assoclist_from_args,
@@ -3813,7 +3813,7 @@
                    [:command,
                     [:@ident, “raise”, [1022, 10]],
                     [:args_add_block,
-                     [[:vcall, [:@ident, “b”, [1022, 16]]]],
+                     [[:var_ref, [:@ident, “b”, [1022, 16]]]],
                      false]]],
                   [:else, [[:var_ref, [:@kw, “true”, [1024, 10]]]]]]]],
                nil,
@@ -3876,7 +3876,7 @@
                      [:@int, “0”, [1033, 15]]],
                     :“&&“,
                     [:binary,
-                     [:vcall, [:@ident, “b”, [1033, 20]]],
+                     [:var_ref, [:@ident, “b”, [1033, 20]]],
                      :==,
                      [:hash, nil]]]],
                   nil]]],
@@ -3946,7 +3946,7 @@
                      [:@int, “0”, [1042, 15]]],
                     :“&&“,
                     [:binary,
-                     [:vcall, [:@ident, “b”, [1042, 20]]],
+                     [:var_ref, [:@ident, “b”, [1042, 20]]],
                      :==,
                      [:hash,
                       [:assoclist_from_args,
@@ -5206,7 +5206,7 @@
                      [[:assoc_new,
                        [:@label, “c:“, [1352, 22]],
                        [:@int, “0”, [1352, 25]]]]]],
-                   [:vcall, [:@ident, “r”, [1352, 29]]]],
+                   [:var_ref, [:@ident, “r”, [1352, 29]]]],
                   false]]],
                [:binary,
                 [:call,
@@ -5299,7 +5299,7 @@
                       [:assoc_new,
                        [:@label, “c:“, [1367, 34]],
                        [:@int, “0”, [1367, 37]]]]]],
-                   [:vcall, [:@ident, “r”, [1367, 41]]]],
+                   [:var_ref, [:@ident, “r”, [1367, 41]]]],
                   false]]],
                [:binary,
                 [:call,
@@ -5931,7 +5931,7 @@
              [:in,
               [:hshptn, nil, [], [:var_field, [:@ident, “r”, [1533, 11]]]],
               [[:binary,
-                [:vcall, [:@ident, “r”, [1534, 8]]],
+                [:var_ref, [:@ident, “r”, [1534, 8]]],
                 :==,
                 [:hash,
                  [:assoclist_from_args,
```
 2024-02-20 17:33:58 +09:00
Nobuyoshi Nakada b1d70e4264
[Bug #20280] Check by `rb_parser_enc_str_coderange` 
Co-authored-by: Yuichiro Kaneko <spiketeika@gmail.com>
 2024-02-19 16:33:26 +09:00
Nobuyoshi Nakada fcc55dc226
[Bug #20280] Raise SyntaxError on invalid encoding symbol 2024-02-19 16:33:26 +09:00
Peter Zhu a71d1ed838 Fix memory leak when parsing invalid hash symbol 
For example:

    10.times do
      100_000.times do
        eval('{"\xC3": 1}')
      rescue EncodingError
      end

      puts `ps -o rss= -p #{$$}`
    end

Before:

    32032
    48464
    66112
    84192
    100592
    117520
    134096
    150656
    167168
    183760

After:

    17120
    17120
    17120
    17120
    18560
    18560
    18560
    18560
    18560
    18560
 2024-02-13 11:05:56 -05:00
yui-knk 33c1e082d0 Remove ruby object from string nodes 
String nodes holds ruby string object on `VALUE nd_lit`.
This commit changes it to `struct rb_parser_string *string`
to reduce dependency on ruby object.
Sometimes these strings are concatenated with other string
therefore string concatenate functions are needed.
 2024-02-09 14:20:17 +09:00
yui-knk 68b57ceb46 Use bool to check ascii only in parse_ident 
No need to use ENC_CODERANGE to record ascii only or not.
 2024-02-03 09:15:41 +09:00
S.H f3df218f48
Introduced `rb_node_const_decl_val` function 
Introduce `rb_node_const_decl_val` function to allow `rb_ary_join` and
`rb_ary_reverse` functions to be removed from Universal Parser.
 2024-01-31 13:31:38 +09:00
Nobuyoshi Nakada e018036d89
Rename `nd_head` in `RNode_RESBODY` as `nd_next` 2024-01-28 11:12:22 +09:00
Nobuyoshi Nakada 0f98d284f3
Remove unused `nd_resq` from `RNode_ENSURE` 2024-01-28 11:11:13 +09:00
S.H 9b40f42c22
Introduce `NODE_ENCODING` 
`__ENCODING__ `was managed by `NODE_LIT` with Encoding object. 

Introduce `NODE_ENCODING` for
1. `__ENCODING__` is detectable from AST Node.
2. Reduce dependency Ruby object for parse.y
 2024-01-27 08:11:10 +00:00
Nobuyoshi Nakada 0610f555ea
Constify `rb_global_parser_config` 2024-01-14 17:55:11 +09:00