This PR fixes an incorrect parsing for `Prism::Translation::Parser`
when one-line pattern mathing with Ruby 2.7 runtime.
## Expected
Parsing should be done based on the specified Ruby parsing version,
independent of the Ruby runtime version. When parsing for Ruby 3.3,
it should return `:match_pattern_p` node:
```console
$ ruby -rprism -rprism/translation/parser33 -ve 'p Prism::Translation::Parser33.parse("foo in bar")'
ruby 3.0.6p216 (2023-03-30 revision https://github.com/ruby/prism/commit/23a532679b) [x86_64-darwin19]
s(:match_pattern_p,
s(:send, nil, :foo),
s(:match_var, :bar))
```
## Actual
When parsing with Ruby 2.7 runtime, `match_pattern` node is returned,
even though it is expected to parse for Ruby 3.3:
```console
$ ruby -rprism -rprism/translation/parser33 -ve 'p Prism::Translation::Parser33.parse("foo in bar")'
ruby 2.7.8p225 (2023-03-30 revision https://github.com/ruby/prism/commit/1f4d455848) [x86_64-darwin19]
s(:match_pattern,
s(:send, nil, :foo),
s(:match_var, :bar))
```
The cause was the use of `RUBY_VERSION` for condition logic,
which made it dependent on runtime Ruby version.
`Prism::Translation::Parser` supports parsing for Ruby 3.3+.
Therefore, the condition for parsing Ruby 2.7, which is not supported, is being removed.
## Background
Found due to incompatibility with RuboCop's `Layout/SpaceAroundKeyword` and `Style/TernaryParentheses` cops.
https://github.com/ruby/prism/commit/e752e251d2
With the introduction of `Prism::Translation::Parser` to RuboCop (RuboCop AST),
the number of arguments for `RuboCop::AST::ProcessedSource#parser_class` internal API will be changed:
https://github.com/rubocop/rubocop-ast/pull/277
## Before
As a result, the following error will occur starting from the next release of RuboCop AST (< 1.30.0) :
```console
$ bundle exec ruby -rrubocop/ast -rprism -rprism/translation/parser/rubocop -ve \
"p RuboCop::AST::ProcessedSource.new('42', 80_82_73_83_77.33).ast"
ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22]
/Users/koic/src/github.com/ruby/prism/lib/prism/translation/parser/rubocop.rb:25:in `parser_class':
wrong number of arguments (given 2, expected 1) (ArgumentError)
from /Users/koic/src/github.com/rubocop/rubocop-ast/lib/rubocop/ast/processed_source.rb:309:in `create_parser'
from /Users/koic/src/github.com/rubocop/rubocop-ast/lib/rubocop/ast/processed_source.rb:219:in `parse'
from /Users/koic/src/github.com/rubocop/rubocop-ast/lib/rubocop/ast/processed_source.rb:47:in `initialize'
from -e:1:in `new'
from -e:1:in `<main>'
```
## After
This PR prevents the above error by updating the monkey patch to support the new argument:
```console
$ bundle exec ruby -rrubocop/ast -rprism -rprism/translation/parser/rubocop -ve \
"p RuboCop::AST::ProcessedSource.new('42', 80_82_73_83_77.33).ast"
ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22]
s(:int, 42)
```
Moreover, to ensure compatibility with the existing RuboCop AST, conditional logic
has been implemented to maintain backward compatibility.
https://github.com/ruby/prism/commit/8a6909f4d5
When creating the Ruby AST, we were previously allocating Location
objects for every node and every inner location. Instead, this
commit changes it to pack both the start offset and length into a
single u64 and pass that into the nodes. Then, when the locations
are requested via a reader method, we lazily allocate the Location
objects.
https://github.com/ruby/prism/commit/de203dca83
Co-Authored-By: Aaron Patterson <tenderlove@ruby-lang.org>
* For Prism.parse_file the file contents would be read as native, then
converted to a Ruby String, then converted to a native String for
pm_serialize_parse().
* Refactor the logic to always use a pm_string for the source code and
pass that to other native functions.
https://github.com/ruby/prism/commit/9002b3c47d
## Summary
Fixes https://github.com/ruby/prism/pull/2356.
I'm working on integrating Prism into RuboCop.
This PR introduces `Prism::Translation::Parser33` and `Prism::Translation::Parser34`, named
in accordance with the following comments.
https://github.com/rubocop/rubocop/issues/12600#issuecomment-1932707748
Currently, `Prism::Translation::Parser` always operates in Ruby 3.4 mode.
This means it will not parse as Ruby 3.3 even if `TargetRubyVersion: 80_82_73_83_77.33` is specified.
Therefore, the `it` introduced in Ruby 3.4 is parsed incompatibly with Ruby 3.3. In Ruby 3.3,
the expected name for an `lvar` is `:it`, not `:"0it"`.
### Expected AST
The following is an expected AST when parsing Ruby 3.3 code:
```console
$ bundle exec ruby -rprism -rprism/translation/parser33 -ve "p Prism::Translation::Parser33.parse('items.map { it.do_something }')"
ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22]
s(:block,
s(:send,
s(:send, nil, :items), :map),
s(:args),
s(:send,
s(:send, nil, :it), :do_something))
```
### Actual AST
The following is an actual AST when parsing Ruby 3.3 code:
```console
$ ruby -rprism -ve "p Prism::Translation::Parser.parse('items.map { it.do_something }')"
ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22]
s(:block,
s(:send,
s(:send, nil, :items), :map),
s(:args),
s(:send,
s(:lvar, :"0it"), :do_something))
```
`Prism::Translation::Parser33` and `Prism::Translation::Parser34` aim to correspond to Ruby 3.3 and Ruby 3.4, respectively.
And, The hack of specifying `TargetRubyVersion: 80_82_73_83_77.33` is expected to become unnecessary in the future,
but the behavior will be maintained until RuboCop's support is finalized:
https://github.com/rubocop/rubocop/issues/12600#issuecomment-1933657732
## Additional Information
A private method named `convert_for_prism` is prepared to convert the `version` from Parser to the `version` expected by Prism.
For example, a Parser-compatible value is `3.3`, whereas Prism expects `"3.3.0"`.
`Parser#version` is not used in RuboCop, but it's unclear how it is utilized in other libraries that rely on the Parser gem.
Therefore, logic to maintain compatibility between Parser and Prism is implemented.
https://github.com/ruby/prism/commit/62d3991e22
LSPs need this because the protocol dictates that you return code
units for offsets. None of our existing APIs provided that
information, and since we hid the source it's not nearly as useful
for them. Now they can pass an encoding directly to:
* Location#start_code_units_offset
* Location#end_code_units_offset
* Location#start_code_units_column
* Location#end_code_units_column
https://github.com/ruby/prism/commit/4757a2cc06
Co-Authored-By: Vinicius Stock <vinicius.stock@shopify.com>
* add bin/prism ripper to compare Ripper output
* block arg handling is quirky, do it per-call-site
* block required params
* boolean values
* various assign-operator support
* breaks, early fragile begin/rescue/end
* more fixtures being checked
https://github.com/ruby/prism/commit/31732cb720
We keep adding more scripts to /bin that are doing the same kinds
of processing. Instead, this commit consolidates them all into a
single CLI that shares the same logic so that we can consistently
read files in the same way.
It keeps around 2 binstubs for bin/lex and bin/parse since those
are the most used and I'm sure people have built up muscle memory
for those. Those scripts are now just wrappers for forwarding to
bin/prism.
https://github.com/ruby/prism/commit/bddcb9bf17
This PR implements proper file parsing error handling. Previously
`file_options` would call `pm_string_mapped_init` which would print an
error from `perror`. However this wouldn't raise a proper Ruby error so
it was just a string output. I've done the following:
- Raise an error from `rb_syserr_fail` with the filepath in
`file_options`.
- No longer return `Qnil` if `file_options` returns false (because now
it will raise)
- Update `file_options` to return `static void` instead of `static
bool`.
- Update `file_options` and `profile_file` to check the type so when
passing `nil` we see a `TypeError`.
- Delete `perror` from `pm_string_mapped_init`
- Update `FFI` backend to raise appropriate errors when calling
`pm_string_mapped_init`.
- Add tests for `dump_file`, `lex_file`, `parse_file`,
`parse_file_comments`, `parse_lex_file`, and `parse_file_success?`
when a file doesn't exist and for `nil`.
- Updates the `bin/parse` script to no longer raise it's own
`ArgumentError` now that we raise a proper error.
Fixes: ruby/prism#2207https://github.com/ruby/prism/commit/b2f7494ff5
Split up the diagnostic levels so that error and warning levels
aren't mixed. Also fix up deconstruct_keys implementation.
https://github.com/ruby/prism/commit/bd3eeb308d
Co-authored-by: Benoit Daloze <eregontp@gmail.com>
It's possible to repeat parameters in method definitions like so:
```ruby
def foo(_a, _a)
end
```
The compiler needs to know to adjust the local table size to account for
these duplicate names. We'll use the repeated parameter flag to account
for the extra stack space required
https://github.com/ruby/prism/commit/b443cb1f60
Co-Authored-By: Kevin Newton <kddnewton@gmail.com>
Co-Authored-By: Jemma Issroff <jemmaissroff@gmail.com>
Because this is a user-facing change, we also need to deal with the
fact that CRuby 3.3.0 was just released.
In order to support workflows that want to parse exactly as CRuby
parses in a specific version, this PR introduces a new option to
the options struct that is "version". This allows you to specify
that you want "3.3.0" parsing.
I'm not sure if this is the correct solution. Another solution is
to just fork and keep around the old branch for security patches.
Or we could keep around a copy of the source files within this
repository as another directory and only update when necessary.
There are a lot of potential solutions here.
Because this change is so small and the check for it is so minimal,
I've decided to go with this enum. If this ends up entirely
cluttering the codebase with version checks, we'll come up with
another solution. But for now this works, so we're going to go in
this direction for a bit until we determine it's no longer working.
https://github.com/ruby/prism/commit/d8c7e6bd10
A lot of tools use Ripper/RubyVM::AbstractSyntaxTree to determine
if a source is valid. These tools both create an AST instead of
providing an API that will return a boolean only.
This new API only creates the C structs, but doesn't bother
reifying them into Ruby/the serialization API. Instead it only
returns true/false, which is significantly more efficient.
https://github.com/ruby/prism/commit/7014740118
Previously numbered parameters were a field on blocks and lambdas
that indicated the maximum number of numbered parameters in either
the block or lambda, respectively. However they also had a
parameters field that would always be nil in these cases.
This changes it so that we introduce a NumberedParametersNode that
goes in place of parameters, which has a single uint8_t maximum
field on it. That field contains the maximum numbered parameter in
either the block or lambda.
As a part of the PR, I'm introducing a new UInt8Field type that
can be used on nodes, which is just to make it a little more
explicit what the maximum values can be (the maximum is actually 9,
since it only goes up to _9). Plus we can do a couple of nice
things in serialization like just read a single byte.
https://github.com/ruby/prism/commit/2d87303903
Fundamentally, `foo { |bar,| }` is different from `foo { |bar, *| }`
because of arity checks. This PR introduces a new node to handle
that, `ImplicitRestNode`, which goes in the `rest` slot of parameter
nodes instead of `RestParameterNode` instances.
This is also used in a couple of other places, namely:
* pattern matching: `foo in [bar,]`
* multi target: `for foo, in bar do end`
* multi write: `foo, = bar`
Now the only splat nodes with a `NULL` value are when you're
forwarding, as in: `def foo(*) = bar(*)`.
https://github.com/ruby/prism/commit/dba2a3b652
* The same order as in source code.
* CallOrWriteNode, CallOperatorWriteNode, CallAndWriteNode already have
the correct order so it was also inconsistent with them.
https://github.com/ruby/prism/commit/4434e4bc22
librubyparser was an artifact of the prototype that was initially
named ruby-parser. Instead, this renames it to libprism to be
consistent with the actual name.
https://github.com/ruby/prism/commit/8600b06811
Prior to this commit, KeywordParameterNode included both optional
and required keywords. With this commit, it is split in two, with
`OptionalKeywordParameterNode`s no longer having a value field.
https://github.com/ruby/prism/commit/89084d9af4
Method calls with keyword splat args compile differently than
without since they merge the keyword arg hash with the keyword splat
hash. We know this information at parse time, so can set a flag
which the compiler can use.
https://github.com/ruby/prism/commit/e5f8a9a3cd
Kevin Newton
Koichi ITO
Gopal Patel
Gopal Patel
Philip Mueller
Noah Gibbs
Max Prokopiev
Benoit Daloze
Vinicius Stock
eileencodes
Aaron Patterson
Adam Hess
Adam Hess
Matthew Healy
Cameron Dutro
Jemma Issroff
Martin Emde
Matt Boldt
Dhaval
heyogrady
Vinicius Stock
Haldun Bayhantopcu
Ryan Garver
Maple Ong
Mateus Pereira
Mau Magnaguagno
Matt Valentine-House
gazayas
BurdetteLamar