Граф коммитов

2103 Коммитов

Автор SHA1 Сообщение Дата
Aaron Patterson d8a4af47a5
Only use `add_mark_object` in Ripper
This patch changes parse.y to only use `add_mark_object` in Ripper.
Previously we were seeing a bug in write barrier verification.  I had
changed `add_mark_object` to execute the write barrier, but the problem
is that we had code like this:

```
NEW_STR(add_mark_object(p, obj), loc)
```

In this case, `add_mark_object` would execute the write barrier between
the ast and `obj`, but the problem is that `obj` isn't actually
reachable from the AST at the time the write barrier executed.
`NEW_STR` can possibly call `malloc` which can kick a GC, and since
`obj` isn't actually reachable from the AST at the time of WB execution,
verification would fail.

Basically the steps were like this:

1. RB_OBJ_WRITTEN via `add_mark_object`
2. Allocate node
3. *Possibly* execute GC via malloc
4. Write obj in to allocated node

This patch changes the steps to:

1. Allocate node
2. *Possibly* execute GC via malloc
3. Write obj in to allocated node
4. RB_OBJ_WRITTEN
2019-09-09 14:26:57 -07:00
Aaron Patterson 4524780d17
Revert "Reverting node marking until I can fix GC problem."
This reverts commit 092f31e7e2.
2019-09-09 14:26:51 -07:00
Yusuke Endoh a3f5265fd1 parse.y: Use the correct alias for brace flag of hash literal
nd_alen and nd_brace is the same field, but nd_brace is more suitable
for this case.
2019-09-08 00:28:03 +09:00
Yusuke Endoh a7a2be7a31 Rename some function/definition names that handles NODE_LIST
from array to list.
Follow up to ac50ac03aeb210763730cdc45f230e236519223d
2019-09-07 13:56:41 +09:00
Yusuke Endoh 99c9431ea1 Rename NODE_ARRAY to NODE_LIST to reflect its actual use cases
and NODE_ZARRAY to NODE_ZLIST.

NODE_ARRAY is used not only by an Array literal, but also the contents
of Hash literals, method call arguments, dynamic string literals, etc.
In addition, the structure of NODE_ARRAY is a linked list, not an array.

This is very confusing, so I believe `NODE_LIST` is a better name.
2019-09-07 13:56:29 +09:00
Nobuyoshi Nakada 3754e15530
Warn local variables which conflict with new numbered parameters 2019-09-06 09:02:12 +09:00
Aaron Patterson 092f31e7e2
Reverting node marking until I can fix GC problem.
Looks like we're getting WB misses during stressful GC on startup.  I am
investigating.
2019-09-05 12:44:23 -07:00
Aaron Patterson 8f096226e1
Stash tmpbuffer inside internal structs
I guess those AST node were actually used for something, so we'd better
not touch them.  Instead this commit just puts the tmpbuffer inside a
different internal struct so that we can mark them.
2019-09-05 11:04:43 -07:00
Aaron Patterson 429ed8d587
Don't change DSTR nodes to ARRAY nodes
DSTR nodes are allocated in to the "markable" bucket where ARRAY nodes
are not.  Switching buckets can cause errors during GC.
2019-09-05 10:13:50 -07:00
Aaron Patterson 545b6db3fb
Create two buckets for allocating NODE structs
This commit adds two buckets for allocating NODE structs, then allocates
"markable" NODE objects from one bucket.  The reason to do this is so
when the AST mark function scans nodes for VALUE objects to mark, we
only scan NODE objects that we know to reference VALUE objects.  If we
*did not* divide the objects, then the mark function spends too much
time scanning objects that don't contain any references.
2019-09-05 10:13:50 -07:00
Aaron Patterson f0fd1c0cd8
Stash the imemo buf at the end of the ID list
Now we can reach the ID table buffer from the id table itself, so when
SCOPE nodes are marked we can keep the buffers alive.  This eliminates
the need for the "mark array" during normal parse / compile (IOW *not*
Ripper).
2019-09-05 10:13:50 -07:00
Aaron Patterson 64817a7cfd
Mark some tmpbufs via node objects
This way we don't need to add the tmpbufs to a Ruby array for marking
2019-09-05 10:13:50 -07:00
Aaron Patterson 581fcde088
Directly mark node objects instead of using a mark array
This patch changes the AST mark function so that it will walk through
nodes in the NODE buffer marking Ruby objects rather than using a mark
array to guarantee liveness.  The reason I want to do this is so that
when compaction happens on major GCs, node objects will have their
references pinned (or possibly we can update them correctly).
2019-09-05 10:13:49 -07:00
Jeremy Evans c6464c44c0 Fix code locations of array node inside hash node when multiple kw splats
This is broken at least since 2.5 (I didn't check earlier versions).
It resulted in failure in test_ast.rb when the tests were added before
the parser change.

Basically, in remove_duplicate_keys, if the node is modified, set
the location information to the previous location information. The
removal of keys should not affect the location in the code.
2019-09-05 09:57:43 -07:00
Jeremy Evans 1d5066efb0 Make m(**{}) mean call without keywords
Previously, **{} was removed by the parser:

```
$ ruby --dump=parse -e '{**{}}'
 @ NODE_SCOPE (line: 1, location: (1,0)-(1,6))
 +- nd_tbl: (empty)
 +- nd_args:
 |   (null node)
 +- nd_body:
     @ NODE_HASH (line: 1, location: (1,0)-(1,6))*
     +- nd_brace: 1 (hash literal)
     +- nd_head:
         (null node)
```

Since it was removed by the parser, the compiler did not know
about it, and `m(**{})` was therefore treated as `m()`.

This modifies the parser to not remove the `**{}`.  A simple
approach for this is fairly simple by just removing a few
lines from the parser, but that would cause two hash
allocations every time it was used.  The approach taken here
modifies both the parser and the compiler, and results in `**{}`
not allocating any hashes in the usual case.

The basic idea is we use a literal node in the parser containing
a frozen empty hash literal.  In the compiler, we recognize when
that is used, and if it is the only keyword present, we just
push it onto the VM stack (no creation of a new hash or merging
of keywords).  If it is the first keyword present, we push a
new empty hash onto the VM stack, so that later keywords can
merge into it.  If it is not the first keyword present, we can
ignore it, since the there is no reason to merge an empty hash
into the existing hash.

Example instructions for `m(**{})`

Before (note ARGS_SIMPLE):

```
== disasm: #<ISeq:<main>@-e:1 (1,0)-(1,7)> (catch: FALSE)
0000 putself                                                          (   1)[Li]
0001 opt_send_without_block       <callinfo!mid:m, argc:0, FCALL|ARGS_SIMPLE>, <callcache>
0004 leave
```

After (note putobject and KW_SPLAT):

```
== disasm: #<ISeq:<main>@-e:1 (1,0)-(1,7)> (catch: FALSE)
0000 putself                                                          (   1)[Li]
0001 putobject                    {}
0003 opt_send_without_block       <callinfo!mid:m, argc:1, FCALL|KW_SPLAT>, <callcache>
0006 leave
```

Example instructions for `m(**h, **{})`

Before and After (no change):

```
== disasm: #<ISeq:<main>@-e:1 (1,0)-(1,12)> (catch: FALSE)
0000 putself                                                          (   1)[Li]
0001 putspecialobject             1
0003 newhash                      0
0005 putself
0006 opt_send_without_block       <callinfo!mid:h, argc:0, FCALL|VCALL|ARGS_SIMPLE>, <callcache>
0009 opt_send_without_block       <callinfo!mid:core#hash_merge_kwd, argc:2, ARGS_SIMPLE>, <callcache>
0012 opt_send_without_block       <callinfo!mid:m, argc:1, FCALL|KW_SPLAT>, <callcache>
0015 leave
```

Example instructions for `m(**{}, **h)`

Before:

```
== disasm: #<ISeq:<main>@-e:1 (1,0)-(1,12)> (catch: FALSE)
0000 putself                                                          (   1)[Li]
0001 putspecialobject             1
0003 newhash                      0
0005 putself
0006 opt_send_without_block       <callinfo!mid:h, argc:0, FCALL|VCALL|ARGS_SIMPLE>, <callcache>
0009 opt_send_without_block       <callinfo!mid:core#hash_merge_kwd, argc:2, ARGS_SIMPLE>, <callcache>
0012 opt_send_without_block       <callinfo!mid:m, argc:1, FCALL|KW_SPLAT>, <callcache>
0015 leave
```

After (basically the same except for the addition of swap):

```
== disasm: #<ISeq:<main>@-e:1 (1,0)-(1,12)> (catch: FALSE)
0000 putself                                                          (   1)[Li]
0001 newhash                      0
0003 putspecialobject             1
0005 swap
0006 putself
0007 opt_send_without_block       <callinfo!mid:h, argc:0, FCALL|VCALL|ARGS_SIMPLE>, <callcache>
0010 opt_send_without_block       <callinfo!mid:core#hash_merge_kwd, argc:2, ARGS_SIMPLE>, <callcache>
0013 opt_send_without_block       <callinfo!mid:m, argc:1, FCALL|KW_SPLAT>, <callcache>
0016 leave
```
2019-09-05 09:57:43 -07:00
Kazuki Tsujimoto 94d6ec1d90
Make pattern matching support **nil syntax 2019-09-01 16:39:34 +09:00
Nobuyoshi Nakada e80a6f65c8
Made :nil static ID 2019-09-01 13:37:28 +09:00
Jeremy Evans 42adc5bc6b Add back missing warning for duplicate keys in splatted hashes
This reverts the changes to parse.y in
a5b37262524ac39d2af13eea174486370a581c23 as they are not actually
needed and cause the warning for duplicate hash keys to not be
emitted.
2019-08-30 12:39:31 -07:00
Jeremy Evans 4d64693c70 Make ripper support **nil syntax
The on_params hook will use :nil as the keyword rest argument.
There is a new on_nokw_param hook as well.

This fixes a type issue in the previous code, where an ID was
passed where a VALUE was the declared type.  The symbol :nil is
passed instead of the id.
2019-08-30 12:39:31 -07:00
Jeremy Evans 6a9ce1fea8 Support **nil syntax for specifying a method does not accept keyword arguments
This syntax means the method should be treated as a method that
uses keyword arguments, but no specific keyword arguments are
supported, and therefore calling the method with keyword arguments
will raise an ArgumentError.  It is still allowed to double splat
an empty hash when calling the method, as that does not pass
any keyword arguments.
2019-08-30 12:39:31 -07:00
Yusuke Endoh 16c6984bb9 Separate keyword arguments from positional arguments
And, allow non-symbol keys as a keyword arugment
2019-08-30 12:39:31 -07:00
Nobuyoshi Nakada 01b723ba6d
Refined warnings against literal in flip-flop 2019-08-29 23:06:27 +09:00
Nobuyoshi Nakada 2ed68d0ff9
Revert "Add pipeline operator [Feature #15799]"
This reverts commits:
* d365fd5a02
* d780c36624
* aa7211836b
* 043f010c28
* bb4dd7c6af05c7821d572e2592ea3d0cc748d81f
* 043f010c28
* f169043d81

http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-core/94645
2019-08-29 15:27:59 +09:00
Aaron Patterson 932a471d38
Directly mark compile options from the AST object
`rb_ast_t` holds a reference to this object, so it should mark the
object.  Currently it is relying on the `mark_ary` on `node_buffer` to
ensure that the object stays alive.  But since the array internals can
move, this could cause a segv if compaction impacts the array.
2019-08-27 11:43:18 -07:00
Kazuhiro NISHIYAMA 5d5502dc85
Fix typos 2019-08-24 21:26:24 +09:00
Nobuyoshi Nakada d5b917d500
Named numbered parameter indexes 2019-08-24 01:10:42 +09:00
Jeremy Evans 53b3be5d58 Fix parsing of mutiple assignment with rescue modifier
Single assignment with rescue modifier applies rescue to the RHS:

  a = raise rescue 1 # a = (raise rescue 1)

Previously, multiple assignment with rescue modifier applied rescue
to the entire expression:

  a, b = raise rescue [1, 2] # (a, b = raise) rescue [1, 2]

This makes multiple assignment with rescue modifier consistent with
single assignment with rescue modifier, applying rescue to the RHS:

  a, b = raise rescue [1, 2] # a, b = (raise rescue [1, 2])

Implements [Feature #8239]
Fixes [Bug #8279]
2019-08-09 09:25:30 -07:00
Takashi Kokubun a3188f43a8
Revert "Revert "Fix dangling path name from fstring""
This reverts commit 326c00b6f8.

We also confirmed that test_gced_eval_location fails without the changes:
https://travis-ci.org/ruby/ruby/builds/567417818
https://rubyci.org/logs/rubyci.s3.amazonaws.com/arch/ruby-master/log/20190804T000003Z.fail.html.gz
https://rubyci.org/logs/rubyci.s3.amazonaws.com/ubuntu1604/ruby-master/log/20190804T003005Z.fail.html.gz
https://rubyci.org/logs/rubyci.s3.amazonaws.com/icc-x64/ruby-master/log/20190804T000007Z.fail.html.gz
https://rubyci.org/logs/rubyci.s3.amazonaws.com/unstable10x/ruby-master/log/20190804T001806Z.fail.html.gz
https://rubyci.org/logs/rubyci.s3.amazonaws.com/debian9/ruby-master/log/20190804T003005Z.fail.html.gz
2019-08-04 10:08:17 +09:00
Takashi Kokubun 326c00b6f8
Revert "Fix dangling path name from fstring"
This reverts commit 5931857281 temporarily,
leaving `TestEval#test_gced_eval_location` to see the impact for missing
the changes.

That's because too many CIs are failing for `require` behaviors:
http://rubyci.s3.amazonaws.com/freebsd11zfs/ruby-master/log/20190803T063004Z.fail.html.gz
http://rubyci.s3.amazonaws.com/unstable10x/ruby-master/log/20190803T051806Z.fail.html.gz
http://rubyci.s3.amazonaws.com/unstable11x/ruby-master/log/20190803T052406Z.fail.html.gz
http://rubyci.s3.amazonaws.com/unstable10s/ruby-master/log/20190803T111909Z.fail.html.gz
http://rubyci.s3.amazonaws.com/unstable11s/ruby-master/log/20190803T062506Z.fail.html.gz
http://rubyci.s3.amazonaws.com/solaris11s-sunc/ruby-master/log/20190803T052505Z.fail.html.gz
https://app.wercker.com/ruby/ruby/runs/mjit-test1/5d4512c921ca08000857936a?step=5d451305c2809c0008a3da76
https://app.wercker.com/ruby/ruby/runs/mjit-test2/5d4512c921ca080008579371?step=5d4513000421020007ca122d
http://ci.rvm.jp/results/trunk_gcc4@silicon-docker/2177591
http://ci.rvm.jp/results/trunk_gcc6@silicon-docker/2177596
http://ci.rvm.jp/results/trunk_clang_60@silicon-docker/2178802
http://ci.rvm.jp/results/trunk-theap-asserts@silicon-docker/2177555
http://ci.rvm.jp/results/trunk-mjit-wait@silicon-docker/2178747

Mostly `TestRequire#test_race_exception` failures, but in ci.rvm.jp
`require` inside rubyspec hangs very often.
2019-08-04 08:52:41 +09:00
Nobuyoshi Nakada 5931857281
Fix dangling path name from fstring
* parse.y (yycompile): make sure in advance that the `__FILE__`
  object shares a fstring, to get rid of dangling path name.
  Fixed up 53e9908d8a.  [Bug #16041]

* vm_eval.c (eval_make_iseq): ditto.
2019-08-03 13:48:29 +09:00
Nobuyoshi Nakada 688a59f8ac
Show the location of `eval` which uses `__FILE__`/`__LINE__` 2019-08-03 11:32:37 +09:00
Yusuke Endoh e9e17cbc05 parse.y: make a warning for __FILE__ in eval by default
[Bug #4352]
2019-08-02 23:17:19 +09:00
Nobuyoshi Nakada d488464106
Use predefined idOr 2019-08-01 16:35:43 +09:00
Nobuyoshi Nakada fd461dea06
Ripper#validate_object: check if the object is hidden 2019-07-19 07:57:17 +09:00
Nobuyoshi Nakada 59d6ce4f4b
Moved RIPPER_DEBUG methods to Ripper from Kernel 2019-07-19 07:50:32 +09:00
Nobuyoshi Nakada 18bce998dd
Fixed build error with RIPPER_DEBUG 2019-07-19 07:50:32 +09:00
Nobuyoshi Nakada a027c4b5b0
Use Qnull instead of 0 and Qundef 2019-07-19 06:23:19 +09:00
Nobuyoshi Nakada 0e23e0c3a0
Adjust indent [ci skip] 2019-07-17 17:04:37 +09:00
Yusuke Endoh 711dfec3fa parse.y (here_document): remove dead code
str is always zero when evaluating the branch.
Found by Coverity Scan.
2019-07-15 14:08:17 +09:00
Yusuke Endoh 9e3971c3ad Add a /* fall through */ comment 2019-07-15 00:29:56 +09:00
Yusuke Endoh 20a3fb3c4b Add a /* fall through */ comment 2019-07-14 21:34:32 +09:00
Nobuyoshi Nakada 4e038a7e64
Revert "parse.y: Deprecate flip-flops"
This reverts commit bae638ad5b.

[Feature #5400]
2019-07-11 14:52:02 +09:00
Nobuyoshi Nakada 49cad67635
Removed duplicate assignment
This `last_state` is set to `lex.state` just before the `switch`
statement, and `token_flush` nor `nextc` never change the state.
2019-07-10 00:20:00 +09:00
Nobuyoshi Nakada 995ae6d529
Check indent of `end` against `else` if present 2019-07-08 17:09:59 +09:00
Nobuyoshi Nakada a13636e756
Message to pipe should end with a newline 2019-07-07 18:18:22 +09:00
Nobuyoshi Nakada d548073f68
Enable indentation warning against `if` just after `else`
```ruby
if false
  puts 'false'
else if true
  puts 'true'
end # -:5: warning: mismatched indentations at 'end' with 'if' at 3
end
```

[Feature #15990]
2019-07-07 15:47:40 +09:00
Nobuyoshi Nakada 789e49dc7e
Renamed column in token_info to indent 2019-07-07 14:43:54 +09:00
Nobuyoshi Nakada d746a41e85
Multiple codepoints are not allowed at single character literal
It has unintentionally passed since 2.5.
2019-07-05 22:39:54 +09:00
Kazuhiro NISHIYAMA fae4489a3a
Fix a typo 2019-07-05 19:00:37 +09:00
Nobuyoshi Nakada f19e048d24
Do not dispatch a nil token in ripper
As a comment token includes the newline, so delayed newline token
just follows it should not be dispatched.  [Bug #11485]

Co-Authored-By: Jeremy Evans <code@jeremyevans.net>
2019-07-04 15:58:47 +09:00
Nobuyoshi Nakada 78a8888c3e
Hoisted out f_rest_marg
* parse.y (f_rest_marg): extract named and unnamed rest parameter
  in parenthesized method arguments, and should not be warned as
  unused.
2019-06-30 10:57:24 +09:00
Nobuyoshi Nakada 27723b699b
Should not warn massign parameters as unused 2019-06-29 18:45:24 +09:00
Nobuyoshi Nakada 1f7cb4bee9
Omit `EXPR_` prefix to show lex_state 2019-06-27 20:31:10 +09:00
Nobuyoshi Nakada 097554855c
Fix ripper fatal
* parse.y (parser_yylex): return END_OF_INPUT at unterminated here
  document instead of an error.  [Bug #15962]
2019-06-27 15:32:03 +09:00
Nobuyoshi Nakada c8e9e0b74b
Fix wrong "void value expression" error
* parse.y (value_expr_check): `then` or `else` only `if` is not a
  void value expression, as the counterpart is evaluated as `nil`.
  [Bug #15932]
2019-06-17 22:30:52 +09:00
Nobuyoshi Nakada 01b3a38043
Fix wrong "void value expression" error
* parse.y (value_expr_check): if either of `then` or `else`
  statements is not a void value expression, the whole `if` is not
  also a void value expression.  [Bug #15932]
2019-06-17 21:44:06 +09:00
Nobuyoshi Nakada a064e46762
Support Bison 3 2019-06-17 03:09:52 +09:00
Nobuyoshi Nakada 53e9908d8a
Fix memory leak
* string.c (str_replace_shared_without_enc): free previous buffer
  before replaced.

* parse.y (gettable): make sure in advance that the `__FILE__`
  object shares a fstring, to get rid of replacement with the
  fstring later.
  TODO: this hack may be needed in other places.

[Bug #15916]

Co-Authored-By: luke-gru (Luke Gruber) <luke.gru@gmail.com>
2019-06-16 23:51:22 +09:00
Nobuyoshi Nakada 5d79054906
Revert github/pull/2230, commit miss 2019-06-15 12:04:16 +09:00
Nobuyoshi Nakada 6fa4c90448
Prefer `enum yytokentype` to int 2019-06-15 11:58:02 +09:00
Josh Cheek b8730f1251
Multiline method chain with leading dot works for blank lines 2019-06-15 11:25:38 +09:00
Josh Cheek 2240de98c0
Remove blank line I accidentally added 2019-06-15 11:25:38 +09:00
Josh Cheek cc180e9371
Rename LINEND to EMPTYLN 2019-06-15 11:25:38 +09:00
Josh Cheek 5af5dd4639
Omg, it works!
I'll rename it and squash this commit later,
just wanted to make sure I couldn't lose it
(took a long time to come up with).
2019-06-15 11:25:13 +09:00
Nobuyoshi Nakada d365fd5a02
An operator is not allowed just after `|>`
https://twitter.com/yukihiro_matz/status/1139454774640726019
2019-06-14 18:17:50 +09:00
Nobuyoshi Nakada aa7211836b
Continue to the next line beginning with a pipeline 2019-06-13 22:19:47 +09:00
Nobuyoshi Nakada 043f010c28
parse.y: moved pipeline to expr
To allow arguments without parentheses.
2019-06-13 22:03:10 +09:00
Nobuyoshi Nakada f169043d81
Add pipeline operator [Feature #15799] 2019-06-13 18:44:32 +09:00
Nobuyoshi Nakada 39ae88ad0d
Refined syntax error messages 2019-06-11 22:31:57 +09:00
Nobuyoshi Nakada 140b8117bd
&. is not allowed inside LHS of massign
https://hackerone.com/reports/605262
2019-06-11 15:22:16 +09:00
Martin Dürst f258137083 Fix grammar of macro name: ECCESSED -> ECCESSIVE
Fix the name of the macro variable introduced in 0872ea5330
from NODE_SPECIAL_EXCESSED_COMMA to NODE_SPECIAL_EXCESSIVE_COMMA.
2019-06-05 14:03:50 +09:00
Yusuke Endoh 0872ea5330 node.h: Avoid a magic number to represent excessed comma
`(ID)1` was assigned to NODE_ARGS#rest_arg for `{|x,| }`.
This change removes the magic number by introducing an explicit macro
variable for it: NODE_SPECIAL_EXCESSED_COMMA.
2019-06-04 23:17:18 +09:00
Nobuyoshi Nakada aa8d393d8a
EOF by 2 ^D on a TTY
Terminate the input from a TTY by 2 ^D at the middle of line, like
as many programs, `cat`, `perl` and so on, do.  By the first ^D,
the line will be sent without a newline, and then EOF will be send
by the next ^D.
2019-06-04 09:24:17 +09:00
Nobuyoshi Nakada c990b3c41a
Fix the error token on "invalid hex escape"
* parse.y (tok_hex): flush token after dispatching the "invalid
  hex escape" parse error.
2019-06-03 15:15:54 +09:00
Nobuyoshi Nakada 92ecf58b1e
parse.y: adjust here-doc error token
* parse.y (here_document): adjust token to the here-doc identifier
  in compile_error when a here-document misses the closing
  identifier.
2019-05-30 22:03:33 +09:00
Nobuyoshi Nakada 4a31c1e45d
parse.y: continue after heredoc error
* parse.y: continue parsing the rest of the here-document starting
  line, after the terminator was not found.
2019-05-30 12:38:50 +09:00
Nobuyoshi Nakada 1da5c73932
parse.y: fix state after ivar/cvar
* parse.y (parse_atmark): return EXPR_END or EXPR_ENDFN, depending
  on the previous state, even incomplete names consistently.
2019-05-29 21:42:58 +09:00
Nobuyoshi Nakada cc66272e50
parse.y: flush invalid char 2019-05-29 13:24:24 +09:00
Nobuyoshi Nakada c730c25354
parse.y: warn escaped whitespace
* parse.y (warn_space_char_code): warn whitespace characters
  escaped with meta/control prefix.
2019-05-29 00:00:26 +09:00
Nobuyoshi Nakada 62b3d4c721
Skip following all digits after `@@` 2019-05-28 14:36:29 +09:00
Nobuyoshi Nakada 1cdaa17a06
parse.y: numbered parameter symbol
* parse.y (parse_atmark): numbered parameter name is not allowed
  as a symbol regardless the context.
2019-05-28 09:00:02 +09:00
Nobuyoshi Nakada b4365e75fd
Do not make an incomplete escape a valid char 2019-05-27 23:08:48 +09:00
Nobuyoshi Nakada c40003da25
Ripper#token
* parse.y (ripper_token): added Ripper#token which returns the
  current token string.  [EXPERIMENTAL]
2019-05-27 23:08:45 +09:00
Nobuyoshi Nakada ea6e284d86
parse.y: removed "parser_" prefix from tokadd_utf8 2019-05-27 13:58:41 +09:00
Nobuyoshi Nakada a43c637660
parse.y: broke the terminator condition down
* parse.y (here_document): broke the terminator condition down
  into each piece, the positional condition, resetting the
  dedented here-document indentation, and matching identifier.
  suppress a false warning by icc.
2019-05-27 00:40:30 +09:00
Nobuyoshi Nakada 2ce6365f9c
parse.y: adjust error indicator
* parse.y (parser_yylex): adjust the error indicator of unexpected
  fraction part.

before:
~~~
1.2.3
^~~
~~~

after:
~~~
1.2.3
   ^~
~~~
2019-05-26 18:47:52 +09:00
Nobuyoshi Nakada 02b39daef8
Fix scanner event at invalid syntax
* parse.y (parser_yyerror, parser_compile_error): revert
  r67224 (e5d10cda07) "Flush erred
  token".
2019-05-26 16:24:33 +09:00
Nobuyoshi Nakada 2893550452
Mixed encoding error can continue to parse 2019-05-24 16:12:17 +09:00
Jeremy Evans c05eaa9325 Fix mixed encoding in heredoc
Heredocs are parsed line-by-line, so we need to keep track of the
temporary encoding of the string.  Previously, a heredoc would
only detect mixed encoding errors if they were on the same line,
this changes things so they will be caught on different lines.

Fixes [Bug #15839]
2019-05-22 21:52:20 -07:00
Nobuyoshi Nakada 7ef548c0ac
Mixed encoding error can continue to parse 2019-05-20 12:43:52 +09:00
Nobuyoshi Nakada 0ed9bdfc13
Pack struct rb_strterm_heredoc_struct on 32-bit platforms 2019-05-16 18:25:16 +09:00
Nobuyoshi Nakada e9bb30d47e
Expect no conflict in the parser 2019-05-12 00:13:52 +09:00
Nobuyoshi Nakada 79931dddba
Propagate parser_params to rb_yytnamerr 2019-05-10 22:02:54 +09:00
Hiroshi SHIBATA aa52464efd
15f45ae4d1 and 56528da3e broke the darwin environment.
Revert "Propagate parser_params to rb_yytnamerr"

This reverts commit 15f45ae4d1.
2019-05-10 21:48:02 +09:00
Nobuyoshi Nakada 15f45ae4d1
Propagate parser_params to rb_yytnamerr 2019-05-10 16:40:34 +09:00
Jeremy Evans bb4ac7a650
Fix use of numbered parameter inside proc that is default value of optarg
This allows cases such as:

```ruby
m ->(a = ->{@1}) {a}
m.call.call(1)

m2 ->(a: ->{@1}) {a}
m2.call.call(2)
```

Previously, this would cause a syntax error.

[Bug#15789]
2019-05-05 16:32:45 +09:00
Jeremy Evans b8f3be295b
Fix a case where numbered parameters should not be allowed
Because `proc{|| @1}` is a syntax error, the following should
also be syntax errors:

```ruby
proc { |
| @1}
```

```ruby
proc { |; a| @1 }
```

This fixes both cases.

[Bug #15825]
2019-05-05 14:46:39 +09:00
Nobuyoshi Nakada ff21e75d32
parse.y: duplicated when clause warning
* parse.y (case_args): moved "duplicated when clause" warning from
  compile phase, so that `ruby -wc` shows them.
2019-05-05 00:29:12 +09:00
Nobuyoshi Nakada 6a3165e19d
Fixed HERETERM_LENGTH_MAX on IL32LLP64 2019-04-29 18:32:21 +09:00
Nobuyoshi Nakada c7f780c155
Refactored rb_strterm_heredoc_t 2019-04-29 17:45:58 +09:00
Nobuyoshi Nakada 1432471a75
Disallow also CR in here-doc identifier
* parse.y (heredoc_identifier): CR in here-document identifier
  might or might not result in a syntax error, by the EOL code.
  make a syntax error regardless of the EOL code.
2019-04-29 13:47:20 +09:00
Nobuyoshi Nakada 330b376133
parse.y: fix here-doc identifier with newline
* parse.y (heredoc_identifier): quoted here-document identifier
  must end within the same line.

  the only corner case that here-document identifier can contain a
  newline is that the closing qoute is placed at the beginning of
  the next line, and has been warned since 2.4.

  ```ruby
  <<"EOS
  " # warning: here document identifier ends with a newline
  EOS
  ```
2019-04-29 12:49:59 +09:00
Nobuyoshi Nakada 54eac83b2a
Hide internal IDs
* parse.y (internal_id): number the ID serial for internal use by
  counting down from the neary maximum value, not to accidentally
  match permanent IDs.

[Bug #15768]
2019-04-26 09:33:48 +09:00
Nobuyoshi Nakada 2272cb00ea
Ripper does not use internal IDs directly 2019-04-25 20:33:18 +09:00
Nobuyoshi Nakada 89271d4a37
Adjusted indents 2019-04-23 20:27:24 +09:00
Seiei Miyagi 6ca9e7cc07
Disallow numbered parameter as the default value of optional argument
[Fix GH-2139] [Bug #15783]
2019-04-23 12:16:15 +09:00
Seiei Miyagi ae07b66aaa
Fix internal error of `->x:@2{}`
[Fix GH-2139] [Bug #15783]
2019-04-23 12:16:14 +09:00
Nobuyoshi Nakada 9cdb736672
Missing semicolon 2019-04-23 10:24:31 +09:00
ktsj 243842f68a Avoid usage of the dummy empty BEGIN node
Use NODE_SPECIAL_NO_NAME_REST instead.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67629 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-04-20 03:37:22 +00:00
nobu c3c4e411eb parse.y: suppress warning
* parse.y (parser_append_options): explicitly pass $; when auto
  splitting, to suppress the warning for non-nil $;.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67608 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-04-18 22:24:03 +00:00
ktsj 9738f96fcf Introduce pattern matching [EXPERIMENTAL]
[ruby-core:87945] [Feature #14912]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67586 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-04-17 06:48:03 +00:00
nobu d8442c4306 parse.y: fix fatal messages
* parse.y (rb_parser_fatal): fix "parser" in the message which was
  replaced accidentally.  it is not the argument name.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67503 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-04-10 12:43:32 +00:00
kazu 25c1fd3b90 Reverting all commits from r67479 to r67496 because of CI failures
Because hard to specify commits related to r67479 only.
So please commit again.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67499 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-04-10 09:15:21 +00:00
nobu cc02df324f parse.y: fix fatal messages
* parse.y (rb_parser_fatal): fix "parser" in the message which was
  replaced accidentally.  it is not the argument name.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67492 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-04-10 05:12:33 +00:00
mame 95f7992b89 Introduce beginless range [Feature#14799]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67422 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-04-03 08:11:41 +00:00
nobu ee5b54822c parse.y: show error line separately
* parse.y: show compile error and the error line separately,
  instead of building the error message by snprintf then yyerror.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67358 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-03-28 10:19:08 +00:00
nobu 740248425c parse.y: make tNUMPARAM id
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67308 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-03-20 00:31:02 +00:00
nobu afa1505ca8 parse.y: removed redundant number_arg parser event
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67307 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-03-19 06:23:36 +00:00
nobu fa66569afa parse.y: fix var_ref of numbered param in ripper
* parse.y (string_dvar, user_variable): register numbered
  parameter in ripper for var_ref.
  [ruby-core:91867] [Bug #15673]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67306 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-03-19 06:01:21 +00:00
nobu e39f7e64b7 parse.y: fix segv with Ripper#yydebug
* parse.y (parser_token_value_print): in ripper, ID values are
  wrapped in NODE_RIPPER at set_yylval_name(), so print the Symbol
  wrapped together.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67305 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-03-19 02:35:50 +00:00
nobu ae5d9a7664 Added Ripper#debug_output
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67304 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-03-19 02:25:15 +00:00
nobu 964bbc1686 parse.y: numbered parameter in lambda
* parse.y (lambda): support numbered parameters, only when no
  argument list including empty parentheses, like empty vertical
  bars.  [ruby-core:91859] [Bug #15672]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67295 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-03-18 12:48:18 +00:00
nobu 17018d4c31 parse.y: continue after invalid name
* parse.y (set_yylval_noname): continue after an invalid global,
  instance, class variable name, without "unexpected end-of-file"
  error.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67292 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-03-18 06:42:19 +00:00
nobu d0e25ed277 parse.y: parser_numbered_param
* parse.y (parser_numbered_param): hoisted out the contextual
  check for numbered parameters.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67289 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-03-18 01:18:25 +00:00
nobu 12acc751e3 Numbered parameters [Feature #4475]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67278 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-03-17 05:21:18 +00:00
ktsj 7006fdecf2 dsym should be treated as string [ruby-core:91852] [Bug #15670]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67270 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-03-16 04:00:14 +00:00
nobu 3134b20a01 Show the source line at an invalid class/instance variable
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67264 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-03-15 00:44:51 +00:00
nobu 9800fc26b0 parse.y: no punctuation instance/class variables
* parse.y (parse_atmark): exclude punctuation follows @ marks,
  whereas it is inclusive after $ mark as some punctuation global
  variables exist.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67254 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-03-14 08:13:46 +00:00
nobu 9c077bbed9 parse.y: show invalid global variable line
* parse.y (parse_gvar): show the source line erred by invalid
  global variable, and indicate the variable including the wrong
  punctuation.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67251 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-03-14 05:57:42 +00:00
nobu ccc933d4a6 Hoisted out ruby_show_error_line
* parse.y (ruby_show_error_line): hoisted out from parser_yyerror.

* parse.y (regx_options): revert r67226 and show the error line
  separately.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67250 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-03-14 05:32:30 +00:00
nobu 86c5806c6a More initializations of YYLTYPE
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67249 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-03-14 02:18:05 +00:00
nobu 8415df619e Remove unnecessary local variable
newline is always "\n" when it is used.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67243 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-03-13 01:44:34 +00:00
nobu b6468b01f7 Show unkwon regexp option line
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67226 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-03-12 04:23:17 +00:00
nobu e5d10cda07 Flush erred token
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67224 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-03-12 04:11:41 +00:00
nobu bda6df356a parse.y: moved excessed_comma
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67216 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-03-11 12:47:02 +00:00
nobu 60023cf452 parse.y: trim newline in erred code
* parse.y (parser_yyerror): trim a newline at the end of the erred
  code which was replaced with an extra space in the succeeding
  cursor line.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66927 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-01-26 13:43:32 +00:00
nobu 6375c68f88 parse.y: function parser_mixed_error & parser_mixed_escape
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66919 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-01-24 12:36:04 +00:00
nobu 7e68efbc95 parse.y: remove an extra error message
* parse.y (parse_string): bail out when word-list meets end of
  input not to show an extra "unexpected" error message after the
  preceding error.

      $ ruby -e "%w["
      -e:1: unterminated string meets end of file
      -e:1: syntax error, unexpected terminator, expecting ' '

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66918 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-01-24 11:43:11 +00:00
nobu c56f61161a parse.y: token name of tSTRING_DEND
* parse.y (rb_yytnamerr): show single-quoted single char as-is, to
  show a quoted closing brace as tSTRING_DEND.  e.g.:

      $ ./ruby -e '"#{true'
      -e:1: syntax error, unexpected end-of-input, expecting '}'

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66911 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-01-23 09:39:45 +00:00
nobu 71342a46ac parse.y: token names
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66910 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-01-23 09:08:22 +00:00
nobu bcf85587b1 parse.y: literal add_mark_object
* parse.y (set_yylval_str, set_yylval_literal): always imply
  add_mark_object.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66907 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-01-22 23:21:58 +00:00
nobu 4e64c54728 ripper.y: get rid of compile error
yystpcpy is always used by yysyntax_error in bison 2.3, but may
not used by other than yytnamerr in newer bison.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66905 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-01-22 07:25:08 +00:00
nobu 59a6215af6 parse.y: enclose keyword token names by quotes
* parse.y (rb_yytnamerr): strip enclosing double-quotes, same as
  the default yytnamerr except for that single-quotes matching
  back-quotes do not stop stripping.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66903 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-01-22 06:24:59 +00:00
nobu 62ec5eb018 parse.y: more token names
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66896 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-01-21 10:24:56 +00:00
nobu 9cbb4dde74 parse.y: refactored symbol rules
* parse.y (symbol): turned into a node, as well as `numeric`, for
  a symbol literal, and includes `dsym` now.

* parse.y (ssym): previous `symbol`.  renamed as the counterpart
  of `dsym`.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66724 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-01-05 13:15:07 +00:00
nobu 59f59f82d4 parse.y: refine error messages
* parse.y: refine error messages for tSYMBEG and tSTRING_BEG, which
  can appear at invalid places.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66722 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-01-05 11:35:24 +00:00
nobu 67c5747369 Method reference operator
Introduce the new operator for method reference, `.:`.
[Feature #12125] [Feature #13581]
[EXPERIMENTAL]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66667 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-12-31 15:00:37 +00:00
nobu f89238ec0d parse.y: ignore constant name captures
* parse.y (reg_named_capture_assign_iter): ignore non-local name
  captures, including non-ASCII constant names.
  [ruby-dev:50719] [Bug #15437]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66463 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-12-20 05:43:15 +00:00
yui-knk dd78257919 Fix locations of NODE_UNDEF in undef with multiple args
* parse.y: Fix the beginning position of trailing NODE_UNDEF.

  e.g. The location of the NODE_UNDEF for `b` is fixed:

  ```
  undef a, b
  ```

  * Before

  ```
  NODE_UNDEF (line: 1, location: (1,6)-(1,10))
  ```

  * After

  ```
  NODE_UNDEF (line: 1, location: (1,9)-(1,10))
  ```

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65973 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-11-25 12:31:35 +00:00
nobu 87b03e8c3c parser: improve error messages
[Fix GH-2011]

From: Akim Demaille <akim.demaille@gmail.com>

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65969 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-11-25 08:31:40 +00:00
nobu f800c5fc8e Constified ruby_sourcefile
which usually refers ruby_sourcefile_string and is not freed
directly.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65655 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-11-10 12:04:18 +00:00
nobu 6e610f5ea7 Parse the source in SCRIPT_LINES__ as array
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65653 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-11-10 11:40:33 +00:00
nobu 4bf84ede20 Get rid of setting SCRIPT_LINES__ by AST.parse
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65647 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-11-09 13:39:36 +00:00
yui-knk ce798d08de Implement `RubyVM::AST.of` [Feature #14836]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65542 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-11-05 02:13:45 +00:00
nobu 0c6edda08c parse.y: last node of NODE_ARRAY
* parse.y (heredoc_dedent): manage the last node of NODE_ARRAY,
  when concatenating dedented literals.
  [ruby-core:89649] [Bug #15272]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65467 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-10-31 02:57:30 +00:00
nobu a769027f79 parse.y: last location from bison
* parse.y (command): set the last location from the location
  managed by bison, so that other nodes are not needed.
  [ruby-core:89648] [Bug #15271]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65460 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-10-30 23:43:13 +00:00
mame c9d720b873 parse.y: remove "shadowing outer local variable" warning
You can now write the following without warning.

    user = User.all.find {|user| cond(user) }

Fixes [Feature #12490].
A patch from Soutaro Matsumoto <matsumoto@soutaro.com>.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65369 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-10-26 03:10:02 +00:00
mame b171d92046 Revert "parse.y: remove "shadowing outer local variable" warning"
I forgot to add the copyright of the patch...

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65368 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-10-26 03:08:10 +00:00
mame 7f69d4e41f parse.y: remove "shadowing outer local variable" warning
You can now write the following without warning.

  user = User.all.find {|user| cond(user) }

Fixes [Feature #12490].

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65367 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-10-26 02:54:01 +00:00
yui-knk 2bfbec8745 parse.y: Fix the compile error
* parse.y: Fix "error: ISO C90 forbids mixed declarations and code [-Werror=declaration-after-statement]"

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65229 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-10-20 11:06:05 +00:00
yui-knk 4d4b60acea Fix locations of NODE_SCOPE in lambda.
* parse.y: Fix to start with the argument.

  e.g. The locations of the NODE_SCOPE is fixed:

  ```
  -> x { 1 + 2 }
  ```

  * Before

  ```
  NODE_SCOPE (line: 1, location: (1,2)-(1,14))
  ```

  * After

  ```
  NODE_SCOPE (line: 1, location: (1,3)-(1,14))
  ```

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65227 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-10-20 10:53:31 +00:00
yui-knk 46447667f1 Fix locations of NODE_LAMBDA.
* node.h: Add `nd_first_loc` and `nd_set_first_loc`

* parse.y: Fix to start with the beginning of `->` .

  e.g. The locations of the NODE_LAMBDA is fixed:

  ```
  -> x { 1 + 2 }
  ```

  * Before

  ```
  NODE_LAMBDA (line: 1, location: (1,2)-(1,14))
  ```

  * After

  ```
  NODE_LAMBDA (line: 1, location: (1,0)-(1,14))
  ```

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65221 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-10-20 10:10:46 +00:00
yui-knk 39fd63428d parse.y (struct ripper_args): Remove not used struct definition
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65198 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-10-20 05:33:37 +00:00
nobu d325d74174 parse.y: fix block passing with empty kwargs
* parse.y (arg_blk_pass): preceeding arguments node may be NULL when
  an empty keyword argument hash splat is optimized away.
  [ruby-core:88890] [Bug #15087]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64786 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-09-20 01:43:35 +00:00
mame fad01941b3 parse.y (arg_append): support NODE_ARGSCAT case
Because of the lack of this case, `[*ary,1,2,3,4,5,6]` was parsed into
an inefficient AST like `ary + [1,2] + [3,4] + [5,6]`.

A patch from Anmol Chopra <anmolchopra@rocketbox.in>.
Fixes [Bug #15018].

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64510 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-08-22 15:04:05 +00:00
mame d65f7458bc parse.y: remove coverage-related code fragments
The code fragments that initializes coverage data were scattered into
both parse.y and compile.c.  parse.y allocated a coverage data, and
compile.c initialize the data.

To remove this cross-cutting concern, this change moves the allocation
from "coverage" function of parse.y to "rb_iseq_new_top" of iseq.c.
For the sake, parse.y just counts the line number of the original source
code, and the number is passed via rb_ast_body_t.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64508 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-08-22 10:38:56 +00:00
yui-knk e70cad6091 parse.y (heredoc_identifier): Update comment for term_len
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63976 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-07-16 02:07:44 +00:00
yui-knk d21f468dab parse.y: Add comment for `rb_strterm_heredoc_struct.sourceline`
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63974 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-07-16 01:26:25 +00:00
mame bae638ad5b parse.y: Deprecate flip-flops
Ref #5400

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63667 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-06-15 08:53:15 +00:00
mame 606d6b3470 Revert "range.c: prohibit `(1..nil)`"
This reverts commit a44c010764a16ae09aaed49d76eec055ca0057c8.
Refs #14845.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63649 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-06-13 11:00:28 +00:00
mame 48de2ea5f9 range.c: prohibit `(1..nil)`
Now endless range can be created by either a literal `(1..)` or explicit
range creation `Range.new(1, nil)`.  [Bug #14845]

This change is intended for "early failure"; for example,
`(1..var).to_a` causes out of memory if `var` is inadvertently nil.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63646 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-06-13 09:04:32 +00:00
yui-knk 744b0bdb7b parse.y: Fix locations of none and mid-rule actions
When an empty rule or a mid-rule action is reduced,
`YYLLOC_DEFAULT` is called with the third parameter to be zero.
If we use `RUBY_SET_YYLLOC_OF_NONE` to set their locations,
sometimes the end position of NODE indicates a blank.
For example, `a.b ;` generates `NODE_CALL (line: 1, location: (1,0)-(1,4))*`,
whose end position indicates a blank.

This is because of the following reasons:

* `NODE_CALL` is created when `primary_value call_op operation2 opt_paren_args` is
  reduced to `method_call`.
* `opt_paren_args` is `none`.
* `yylex` is called and `lex.pbeg` moves before `none` is reduced, so
  the beginning position of `none` does not match with the end position of `operation2`.

To fix locations, use `YYRHSLOC(Rhs, 0)` in `YYLLOC_DEFAULT`
(0 "refers to the symbol just before the reduction").

By this change, the bottom of the location stack would be referenced,
so initialize the bottom with `RUBY_SET_YYLLOC_OF_NONE` in `%initial-action`.

Ref: https://www.gnu.org/software/bison/manual/html_node/Location-Default-Action.html#Location-Default-Action

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63623 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-06-10 06:22:15 +00:00
yui-knk aee5c4338b parse.y: Fix locations of NODE_RESCUE
* parse.y (new_bodystmt): Fix locations of NODE_RESCUE
  to end with nd_else or nd_resq. Before this commit,
  locations of NODE_RESCUE included locations of nd_ensr
  of NODE_ENSURE which is a parent node of NODE_RESCUE.

  e.g. The location of the end of NODE_RESCUE is fixed:

  ```
  def a
    :b
  rescue
    :c
  ensure
    :d
  end
  ```

  * Before

  ```
  NODE_RESCUE (line: 2, location: (2,2)-(6,4))
  ```

  * After

  ```
  NODE_RESCUE (line: 3, location: (2,2)-(5,0))
  ```

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63621 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-06-10 05:02:26 +00:00
normal 1c1ff52f43 parse.y: use ruby_sized_xfree and SIZED_REALLOC_N
Another part of the plan to reduce dependencies on malloc_usable_size
which costs us speed: https://bugs.ruby-lang.org/issues/10238

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63484 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-05-22 00:44:22 +00:00
shyouhei 687308cf0d explicit cast to void* required for %p
No automatic type promotion is expected for variadic arguments.
You have to do it by hand.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63474 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-05-21 00:41:57 +00:00
nobu 4dc5b6ff58 parse.y: relax literal in condition warnings
* parse.y (cond0): do not warn literal boolean (true and false) in
  condition expressions, as they are often used as infinite loops,
  deactivated code block, etc.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63457 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-05-18 00:36:24 +00:00
nobu 8487c63fa3 rb_parser_printf declaration
* node.h (rb_parser_printf): delcare as PRINTF_ARGS.

* parse.y (parser_token_value_print): cast to adjust to %c.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63454 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-05-17 12:36:33 +00:00
nobu b936115ed6 fix potential memory leaks
* parse.y (primary, new_args_tail, local_tbl): keep the order;
  allocate an empty imemo first then xmalloc, to get rid of
  potential memory leak when allocation imemo failed.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63397 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-05-11 13:01:36 +00:00
yui-knk 5afa84b028 parse.y: Fix locations of NODE_NIL in endless ranges.
* parse.y: Fix to start with the end of dots.

  e.g. The locations of the NODE_NIL is fixed:

  ```
  1..
  ```

  * Before

  ```
  NODE_NIL (line: 1, location: (1,0)-(1,3))
  ```

  * After

  ```
  NODE_NIL (line: 1, location: (1,3)-(1,3))
  ```

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63391 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-05-10 06:34:20 +00:00
nobu 6e06d6440b fix potential memory leaks
* gc.c (rb_alloc_tmp_buffer_with_count): keep the order; allocate
  an empty imemo first then xmalloc, to get rid of potential
  memory leak when allocation imemo failed.

* parse.y (rb_parser_malloc, rb_parser_calloc, rb_parser_realloc):
  ditto.

* process.c (rb_execarg_allocate_dup2_tmpbuf): ditto.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63385 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-05-09 17:40:04 +00:00
mame e9303e1a00 Rename imemo_alloc with imemo_tmpbuf
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63372 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-05-09 07:11:59 +00:00
mame af0696782d gc.c (rb_imemo_alloc_new): split for each purpose
imemo_alloc is used for three purposes: auto-free pointer (alternative
of alloca), alloc_tmp_buffer, and heap allocation for bison.
To make it clear, this change introduces three functions:
rb_imemo_alloc_auto_free_pointer,
rb_imemo_alloc_auto_free_maybe_mark_buffer, and
rb_imemo_alloc_parser_heap.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63371 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-05-09 07:08:53 +00:00
mame 58823392b4 gc.c (rb_imemo_alloc_new): improve the API interface
rb_imemo_alloc_new returns rb_imemo_alloc_t*, but took VALUEs, which is
inconsistent.  To make the intention clear, it now takes only a pointer
to the buffer.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63369 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-05-09 06:12:17 +00:00
mame 7f95eed19e Introduce endless range [Feature#12912]
Typical usages:
```
p ary[1..]          # drop the first element; identical to ary[1..-1]
(1..).each {|n|...} # iterate forever from 1; identical to 1.step{...}
```

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63192 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-04-19 15:18:50 +00:00
nobu f168dbd9ef parse.y: fix cmdarg in command_args
* parse.y (call_args): fix invalid CMDARG state after command_args
  followed by tLBRACE_ARG.  [ruby-core:86551] [Bug #14690]

From: Ilya Bylich <ibylich@gmail.com>

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63168 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-04-17 01:22:26 +00:00
nobu 3b93a8bcad parse.y: `else` indent
* parse.y (keyword_else): warn less-indented `else` than `case`.
  [ruby-core:86492] [Bug #14674]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63165 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-04-16 08:54:03 +00:00
nobu a3a5f38d09 parse.y: `when` indent
* parse.y (k_when): warn less-indented `when` than `case`.
  [ruby-core:86492] [Bug #14674]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63141 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-04-12 22:13:09 +00:00
nobu bbcab8c6f3 parse.y: [DOC] Ripper.dedent_string is internal
* parse.y (parser_dedent_string): stated that Ripper.dedent_string
  is for internal use only.

[ci skip]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63127 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-04-09 12:39:59 +00:00
nobu 6a57127222 parse.y: extra error message after no digits
* parse.y (no_digits): return tINTEGER instead of unexpected
  end-of-input, to get rid of extra error messages.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63126 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-04-09 11:09:30 +00:00
stomar d33ddd631b parse.y: fix docs for Ripper.dedent_string
* parse.y: [DOC] fix return type in call-seq of Ripper.dedent_string,
  clarify the method's behavior.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63121 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-04-08 19:39:38 +00:00
nobu 9c5d90b516 Use only CMDARG/COND _PUSH/POP for cmdarg/cond management.
From: Ilya Bylich <ibylich@gmail.com>

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63033 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-03-29 03:42:32 +00:00
nobu e5a30879f9 parse.y: remove `p->` from bitstack traces
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63032 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-03-29 01:33:25 +00:00
nobu f3fbf1dfad parse.y: k_else in bodystmt
* parse.y (bodystmt): expand opt_else to show the error message at
  the right place.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62902 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-03-23 01:09:57 +00:00
nobu 140512d222 parse.y: else without rescue
* parse.y (bodystmt): [EXPERIMENTAL] make `else` without `rescue`
  a syntax error.  [DevelopersMeeting20180315Japan]

  https://bugs.ruby-lang.org/projects/ruby/wiki/DevelopersMeeting20180315Japan
  https://docs.google.com/document/d/1RT0ijSo8uJ4Awn3CEvuYkjH0TVeXSYgeAFNmVGYC3ak/edit#
  > * do-else-end
  >   https://twitter.com/joker1007/status/974173396006129664
  >   https://twitter.com/joker1007/status/974173641347756032
  >   https://twitter.com/joker1007/status/974176512554369027
  >   Will be SyntaxError in 2.6-preview2
  >   All of begin/do/def (experimental)

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62901 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-03-23 00:40:08 +00:00
nobu 721b55d70c parse.y: reorder in the place
* parse.y (new_args_tail): reorder required and optional keyword
  argument variable IDs in the place.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62887 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-03-22 06:47:53 +00:00
nobu 58fbe69a5b parse.y: terminator at continued line
* parse.y (here_document): a continuing line is not the
  terminator.  [ruby-core:86283] [Bug #14621]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62873 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-03-21 12:40:16 +00:00
nobu 491f523ab1 parse.y: unindent continued line
* parse.y (tokadd_string): stop at continued line in dedented here
  documents, to dedent for each lines before removing escaped
  newlines.  [ruby-core:86236] [Bug #14621]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62872 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-03-21 10:32:15 +00:00
nobu b4b3de3e7c node.c: predicates for special NODEs
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62861 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-03-20 13:30:57 +00:00
nobu 6916130566 parse.y: NUL-terminate ruby_sourcefile
* parse.y (yycompile): in some cases (warning, error, dtrace,...),
  ruby_sourcefile is expected to be NUL-terminated, so ensure it.

* template/prelude.c.tmpl (prelude_name): NUL-terminate to get rid
  of copying static data.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62841 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-03-19 14:12:00 +00:00
nobu 7e199e4156 parse.y: push `do` token info
* parse.y (k_rescue, k_ensure): revert r62838.

* parse.y (k_do, k_do_LAMBDA, k_do_block): push token infos of
  `do`.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62839 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-03-19 12:40:34 +00:00