If there are many searches in the `$LOAD_PATH` in the user environment,
require will perform unnecessary searches that are not needed.
In contrast, `require_relative` is efficient because it uses a relative path.
https://github.com/ruby/prism/commit/438ccc67bd
This PR fixes token incompatibility for `Prism::Translation::Parser::Lexer`
when using escaped backslash in string literal:
```ruby
"\\ foo \\ bar"
```
## Parser gem (Expected)
```console
$ bundle exec ruby -Ilib -rparser/ruby33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = File.read("example.rb"); p Parser::Ruby33.new.tokenize(buf)'
ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22]
[s(:str, "\\ foo \\ bar"), [], [[:tSTRING, ["\\ foo \\ bar", #<Parser::Source::Range example.rb 0...15>]],
[:tNL, [nil, #<Parser::Source::Range example.rb 15...16>]]]]
```
## `Prism::Translation::Parser` (Actual)
Previously, the tokens returned by the Parser gem were different. The escaped backslash does not match in the `tSTRING` token:
```console
$ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = File.read("example.rb"); p Prism::Translation::Parser33.new.tokenize(buf)'
ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22]
[s(:str, "\\ foo \\ bar"), [], [[:tSTRING, ["\\\\ foo \\\\ bar", #<Parser::Source::Range example.rb 0...15>]],
[:tNL, [nil, #<Parser::Source::Range example.rb 15...16>]]]]
```
After this correction, the AST and tokens returned by the Parser gem are the same:
```console
$ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = File.read("example.rb"); p Prism::Translation::Parser33.new.tokenize(buf)'
ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22]
[s(:str, "\\ foo \\ bar"), [], [[:tSTRING, ["\\ foo \\ bar", #<Parser::Source::Range example.rb 0...15>]],
[:tNL, [nil, #<Parser::Source::Range example.rb 15...16>]]]]
```
The reproduction test is based on the following strings.txt and exists:
https://github.com/ruby/prism/blob/v0.24.0/test/prism/fixtures/strings.txt#L79
But, the restoration has not yet been performed due to remaining other issues in strings.txt.
https://github.com/ruby/prism/commit/2c44e7e307
In practice, the `BACKTICK` is mapped either as `:tXSTRING_BEG` or `:tBACK_REF2`.
The former is used in xstrings like `` `foo` ``, while the latter is utilized as
a back reference in contexts like `` A::` ``.
This PR will make corrections to differentiate the use of `BACKTICK`.
This mistake was discovered through the investigation of xstring.txt file.
The PR will run tests from xstring.txt file except for `` `f\oo` ``, which will still fail,
hence it will be separated into xstring_with_backslash.txt file.
This separation will facilitate addressing the correction at a different time.
https://github.com/ruby/prism/commit/49ad8df40a
This PR fixes the following error for `Prism::Translation::Parser::Lexer` on the main branch:
```console
$ cat example.rb
'a' # aあ
"
#{x}
"
$ bundle exec rubocop
Parser::Source::Range: end_pos must not be less than begin_pos
/Users/koic/.rbenv/versions/3.0.4/lib/ruby/gems/3.0.0/gems/parser-3.3.0.5/lib/parser/source/range.rb:39:in `initialize'
/Users/koic/src/github.com/ruby/prism/lib/prism/translation/parser/lexer.rb:299:in `new'
/Users/koic/src/github.com/ruby/prism/lib/prism/translation/parser/lexer.rb:299:in `block in to_a'
/Users/koic/src/github.com/ruby/prism/lib/prism/translation/parser/lexer.rb:297:in `map'
/Users/koic/src/github.com/ruby/prism/lib/prism/translation/parser/lexer.rb:297:in `to_a'
/Users/koic/src/github.com/ruby/prism/lib/prism/translation/parser.rb:263:in `build_tokens'
/Users/koic/src/github.com/ruby/prism/lib/prism/translation/parser.rb:92:in `tokenize'
```
This change was made in https://github.com/ruby/prism/pull/2557, and it seems there was
an inconsistency in Range due to forgetting to apply `offset_cache` to `start_offset`.
Prism has been directly supported as a parser engine since RuboCop 1.62:
https://github.com/rubocop/rubocop/releases/tag/v1.62.0
This makes specifying `TargetRubyVersion` with special values like `80_82_73_83_77.33`
using the `prism/translation/parser/rubocop` file unnecessary.
As a result, it would be possible to deprecate this approach.
OTOH, early users might be surprised if `prism/translation/parser/rubocop` were to be suddenly removed.
Therefore, this PR deprecates the parameters related to `prism/translation/parser/rubocop`.
```console
$ bundle exec ruby -rrubocop -rprism/translation/parser/rubocop -e "RuboCop::AST::ProcessedSource.new('42', 80_82_73_83_77.33).ast"
WARN: Prism is directly supported since RuboCop 1.62. The `prism/translation/parser/rubocop` file is deprecated.
WARN: Setting `TargetRubyVersion: 80_82_73_83_77.33` is deprecated. Set to `ParserEngine: parser_prism` and `TargetRubyVersion: 3.3` instead.
$ bundle exec ruby -rrubocop -rprism/translation/parser/rubocop -e "RuboCop::AST::ProcessedSource.new('42', 80_82_73_83_77.34).ast"
WARN: Prism is directly supported since RuboCop 1.62. The `prism/translation/parser/rubocop` file is deprecated.
WARN: Setting `TargetRubyVersion: 80_82_73_83_77.34` is deprecated. Set to `ParserEngine: parser_prism` and `TargetRubyVersion: 3.4` instead.
```
Eventually, it will be possible to remove it at some point.
Regarding documentation, it has been updated to not show the old, discouraged usage but rather
the new way of specifying it in RuboCop.
https://github.com/ruby/prism/commit/0e4bc31463
This PR fixes the following incompatibility AST for regexp match between Parser gem and Prism:
## Parser gem
Returns an `match_with_lvasgn` node:
```console
$ bundle exec ruby -rparser/ruby33 -ve 'p Parser::Ruby33.parse("/foo/ =~ bar")'
ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22]
s(:match_with_lvasgn,
s(:regexp,
s(:str, "foo"),
s(:regopt)),
s(:send, nil, :bar))
```
## Prism (`Prism::Translation::Parser`)
### Before
Returns an `send` node:
```console
$ bundle exec ruby -rprism -rprism/translation/parser33 -ve 'p Prism::Translation::Parser33.parse("/foo/ =~ bar")'
ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22]
s(:send,
s(:regexp,
s(:str, "foo"),
s(:regopt)), :=~,
s(:send, nil, :bar))
```
### After
Returns an `match_with_lvasgn` node:
```console
$ bundle exec ruby -rprism -rprism/translation/parser33 -ve 'p Prism::Translation::Parser33.parse("/foo/ =~ bar")'
ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22]
s(:match_with_lvasgn,
s(:regexp,
s(:str, "foo"),
s(:regopt)),
s(:send, nil, :bar))
```
## Background
Found due to incompatibility with RuboCop's `Performance/EndWith`, `Performance/StringInclude,
and `Performance/StartWith` cops.
## Note
This is the incompatibility when the receiver is a regular expression literal and `=~` is used.
Based on the node name `:match_with_lvasgn`, it appears that Prism's AST becomes more accurate
in cases like `visit_match_write_node` only.
However, as shown in the background, the current behavior of Parser gem is not like this.
Considering compatibility with the published AST of Parser gem, the AST incompatibility will be addressed.
This lvar-injecting feature appears to have not been supported by Parser gem for a long time:
https://github.com/whitequark/parser/issues/69#issuecomment-19506391
There seems to be no indication that it will be supported.
This PR prioritizes AST compatibility between the Parser gem and Prism.
However, it is unclear whether this is the best approach.
https://github.com/ruby/prism/commit/dff4abb170
This PR fixes an incorrect parsing for `Prism::Translation::Parser`
when one-line pattern mathing with Ruby 2.7 runtime.
## Expected
Parsing should be done based on the specified Ruby parsing version,
independent of the Ruby runtime version. When parsing for Ruby 3.3,
it should return `:match_pattern_p` node:
```console
$ ruby -rprism -rprism/translation/parser33 -ve 'p Prism::Translation::Parser33.parse("foo in bar")'
ruby 3.0.6p216 (2023-03-30 revision https://github.com/ruby/prism/commit/23a532679b) [x86_64-darwin19]
s(:match_pattern_p,
s(:send, nil, :foo),
s(:match_var, :bar))
```
## Actual
When parsing with Ruby 2.7 runtime, `match_pattern` node is returned,
even though it is expected to parse for Ruby 3.3:
```console
$ ruby -rprism -rprism/translation/parser33 -ve 'p Prism::Translation::Parser33.parse("foo in bar")'
ruby 2.7.8p225 (2023-03-30 revision https://github.com/ruby/prism/commit/1f4d455848) [x86_64-darwin19]
s(:match_pattern,
s(:send, nil, :foo),
s(:match_var, :bar))
```
The cause was the use of `RUBY_VERSION` for condition logic,
which made it dependent on runtime Ruby version.
`Prism::Translation::Parser` supports parsing for Ruby 3.3+.
Therefore, the condition for parsing Ruby 2.7, which is not supported, is being removed.
## Background
Found due to incompatibility with RuboCop's `Layout/SpaceAroundKeyword` and `Style/TernaryParentheses` cops.
https://github.com/ruby/prism/commit/e752e251d2
With the introduction of `Prism::Translation::Parser` to RuboCop (RuboCop AST),
the number of arguments for `RuboCop::AST::ProcessedSource#parser_class` internal API will be changed:
https://github.com/rubocop/rubocop-ast/pull/277
## Before
As a result, the following error will occur starting from the next release of RuboCop AST (< 1.30.0) :
```console
$ bundle exec ruby -rrubocop/ast -rprism -rprism/translation/parser/rubocop -ve \
"p RuboCop::AST::ProcessedSource.new('42', 80_82_73_83_77.33).ast"
ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22]
/Users/koic/src/github.com/ruby/prism/lib/prism/translation/parser/rubocop.rb:25:in `parser_class':
wrong number of arguments (given 2, expected 1) (ArgumentError)
from /Users/koic/src/github.com/rubocop/rubocop-ast/lib/rubocop/ast/processed_source.rb:309:in `create_parser'
from /Users/koic/src/github.com/rubocop/rubocop-ast/lib/rubocop/ast/processed_source.rb:219:in `parse'
from /Users/koic/src/github.com/rubocop/rubocop-ast/lib/rubocop/ast/processed_source.rb:47:in `initialize'
from -e:1:in `new'
from -e:1:in `<main>'
```
## After
This PR prevents the above error by updating the monkey patch to support the new argument:
```console
$ bundle exec ruby -rrubocop/ast -rprism -rprism/translation/parser/rubocop -ve \
"p RuboCop::AST::ProcessedSource.new('42', 80_82_73_83_77.33).ast"
ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22]
s(:int, 42)
```
Moreover, to ensure compatibility with the existing RuboCop AST, conditional logic
has been implemented to maintain backward compatibility.
https://github.com/ruby/prism/commit/8a6909f4d5
When creating the Ruby AST, we were previously allocating Location
objects for every node and every inner location. Instead, this
commit changes it to pack both the start offset and length into a
single u64 and pass that into the nodes. Then, when the locations
are requested via a reader method, we lazily allocate the Location
objects.
https://github.com/ruby/prism/commit/de203dca83
Co-Authored-By: Aaron Patterson <tenderlove@ruby-lang.org>
## Summary
Fixes https://github.com/ruby/prism/pull/2356.
I'm working on integrating Prism into RuboCop.
This PR introduces `Prism::Translation::Parser33` and `Prism::Translation::Parser34`, named
in accordance with the following comments.
https://github.com/rubocop/rubocop/issues/12600#issuecomment-1932707748
Currently, `Prism::Translation::Parser` always operates in Ruby 3.4 mode.
This means it will not parse as Ruby 3.3 even if `TargetRubyVersion: 80_82_73_83_77.33` is specified.
Therefore, the `it` introduced in Ruby 3.4 is parsed incompatibly with Ruby 3.3. In Ruby 3.3,
the expected name for an `lvar` is `:it`, not `:"0it"`.
### Expected AST
The following is an expected AST when parsing Ruby 3.3 code:
```console
$ bundle exec ruby -rprism -rprism/translation/parser33 -ve "p Prism::Translation::Parser33.parse('items.map { it.do_something }')"
ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22]
s(:block,
s(:send,
s(:send, nil, :items), :map),
s(:args),
s(:send,
s(:send, nil, :it), :do_something))
```
### Actual AST
The following is an actual AST when parsing Ruby 3.3 code:
```console
$ ruby -rprism -ve "p Prism::Translation::Parser.parse('items.map { it.do_something }')"
ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22]
s(:block,
s(:send,
s(:send, nil, :items), :map),
s(:args),
s(:send,
s(:lvar, :"0it"), :do_something))
```
`Prism::Translation::Parser33` and `Prism::Translation::Parser34` aim to correspond to Ruby 3.3 and Ruby 3.4, respectively.
And, The hack of specifying `TargetRubyVersion: 80_82_73_83_77.33` is expected to become unnecessary in the future,
but the behavior will be maintained until RuboCop's support is finalized:
https://github.com/rubocop/rubocop/issues/12600#issuecomment-1933657732
## Additional Information
A private method named `convert_for_prism` is prepared to convert the `version` from Parser to the `version` expected by Prism.
For example, a Parser-compatible value is `3.3`, whereas Prism expects `"3.3.0"`.
`Parser#version` is not used in RuboCop, but it's unclear how it is utilized in other libraries that rely on the Parser gem.
Therefore, logic to maintain compatibility between Parser and Prism is implemented.
https://github.com/ruby/prism/commit/62d3991e22
Koichi ITO
Kevin Newton
Ufuk Kayserilioglu
Max Prokopiev