17e7219679
ext/ripper/lib/ripper/lexer.rb: Do not deprecate Ripper::Lexer::State#[]
...
The old code of IRB still uses this method. The warning is noisy on
rails console.
In principle, Ruby 3.1 deprecates nothing, so let's avoid the
deprecation for the while.
I think It is not so hard to continue to maintain it as it is a trivial
shim.
https://github.com/ruby/ruby/pull/5093
2021-12-09 00:30:17 +09:00
524a808d23
Define Ripper::Lexer::Elem#to_s
...
Alias `#inspect` as `#to_s` also in the new `Ripper::Lexer::Elem`
class, so that `puts Ripper::Lexer.new(code).scan` shows the
attributes.
2021-12-02 18:29:45 +09:00
8944009be7
Deprecate `Lexer::Elem#[]` and `Lexer::State#[]`
...
Discussed in https://github.com/ruby/ruby/pull/5093#issuecomment-964426481 .
> it would be enough to mimic only [] for almost all cases
This adds back the `Lexer::Elem#[]` and `Lexer::State#[]` and adds deprecation warnings for them.
2021-12-02 15:55:42 +09:00
3685b5af95
Only iterate Lexer heredoc arrays
...
The last element in the `@buf` may be either an array or an `Elem`. In the case it is an `Elem` we iterate over every element, when we do not need to. This check guards that case by ensuring that we only iterate over an array of elements.
2021-12-02 15:55:42 +09:00
3f74eaa7a8
~1.10x faster Change Ripper.lex structs to classes
...
## Concept
I am proposing we replace the Struct implementation of data structures inside of ripper with real classes.
This will improve performance and the implementation is not meaningfully more complicated.
## Example
Struct versus class comparison:
```ruby
Elem = Struct.new(:pos, :event, :tok, :state, :message) do
def initialize(pos, event, tok, state, message = nil)
super(pos, event, tok, State.new(state), message)
end
# ...
def to_a
a = super
a.pop unless a.empty?
a
end
end
class ElemClass
attr_accessor :pos, :event, :tok, :state, :message
def initialize(pos, event, tok, state, message = nil)
@pos = pos
@event = event
@tok = tok
@state = State.new(state)
@message = message
end
def to_a
if @message
[@pos, @event, @tok, @state, @message]
else
[@pos, @event, @tok, @state]
end
end
end
# stub state class creation for now
class State; def initialize(val); end; end
```
## MicroBenchmark creation
```ruby
require 'benchmark/ips'
require 'ripper'
pos = [1, 2]
event = :on_nl
tok = "\n".freeze
state = Ripper::EXPR_BEG
Benchmark.ips do |x|
x.report("struct") { Elem.new(pos, event, tok, state) }
x.report("class ") { ElemClass.new(pos, event, tok, state) }
x.compare!
end; nil
```
Gives ~1.2x faster creation:
```
Warming up --------------------------------------
struct 263.983k i/100ms
class 303.367k i/100ms
Calculating -------------------------------------
struct 2.638M (± 5.9%) i/s - 13.199M in 5.023460s
class 3.171M (± 4.6%) i/s - 16.078M in 5.082369s
Comparison:
class : 3170690.2 i/s
struct: 2638493.5 i/s - 1.20x (± 0.00) slower
```
## MicroBenchmark `to_a` (Called by Ripper.lex for every element)
```ruby
require 'benchmark/ips'
require 'ripper'
pos = [1, 2]
event = :on_nl
tok = "\n".freeze
state = Ripper::EXPR_BEG
struct = Elem.new(pos, event, tok, state)
from_class = ElemClass.new(pos, event, tok, state)
Benchmark.ips do |x|
x.report("struct") { struct.to_a }
x.report("class ") { from_class.to_a }
x.compare!
end; nil
```
Gives 1.46x faster `to_a`:
```
Warming up --------------------------------------
struct 612.094k i/100ms
class 893.233k i/100ms
Calculating -------------------------------------
struct 6.121M (± 5.4%) i/s - 30.605M in 5.015851s
class 8.931M (± 7.9%) i/s - 44.662M in 5.039733s
Comparison:
class : 8930619.0 i/s
struct: 6121358.9 i/s - 1.46x (± 0.00) slower
```
## MicroBenchmark data access
```ruby
require 'benchmark/ips'
require 'ripper'
pos = [1, 2]
event = :on_nl
tok = "\n".freeze
state = Ripper::EXPR_BEG
struct = Elem.new(pos, event, tok, state)
from_class = ElemClass.new(pos, event, tok, state)
Benchmark.ips do |x|
x.report("struct") { struct.pos[1] }
x.report("class ") { from_class.pos[1] }
x.compare!
end; nil
```
Gives ~1.17x faster data access:
```
Warming up --------------------------------------
struct 1.694M i/100ms
class 1.868M i/100ms
Calculating -------------------------------------
struct 16.149M (± 6.8%) i/s - 81.318M in 5.060633s
class 18.886M (± 2.9%) i/s - 95.262M in 5.048359s
Comparison:
class : 18885669.6 i/s
struct: 16149255.8 i/s - 1.17x (± 0.00) slower
```
## Full benchmark integration of this inside of Ripper.lex
Inside of this repo with this commit
```
$ cd ext/ripper
$ make
$ cat test.rb
file = File.join(__dir__, "../../array.rb")
source = File.read(file)
bench = Benchmark.measure do
10_000.times.each do
Ripper.lex(source)
end
end
puts bench
```
Then execute with and without this change 50 times:
```
rm new.txt
rm old.txt
for i in {0..50}
do
`ruby -Ilib -rripper -rbenchmark ./test.rb >> new.txt`
`ruby -rripper -rbenchmark ./test.rb >> old.txt`
done
```
I used derailed benchmarks internals to compare the results:
```
dir = Pathname(".")
branch_info = {}
branch_info["old"] = { desc: "Struct lex", time: Time.now, file: dir.join("old.txt"), name: "old" }
branch_info["new"] = { desc: "Class lex", time: Time.now, file: dir.join("new.txt"), name: "new" }
stats = DerailedBenchmarks::StatsFromDir.new(branch_info)
stats.call.banner
```
Which gave us:
```
❤️ ❤️ ❤️ (Statistically Significant) ❤️ ❤️ ❤️
[new] (3.3139 seconds) "Class lex" ref: "new"
FASTER 🚀 🚀 🚀 by:
1.1046x [older/newer]
9.4700% [(older - newer) / older * 100]
[old] (3.6606 seconds) "Struct lex" ref: "old"
Iterations per sample:
Samples: 51
Test type: Kolmogorov Smirnov
Confidence level: 99.0 %
Is significant? (max > critical): true
D critical: 0.30049534876137013
D max: 0.9607843137254902
Histograms (time ranges are in seconds):
[new] description: [old] description:
"Class lex" "Struct lex"
┌ ┐ ┌ ┐
[3.0, 3.3) ┤▇ 1 [3.0, 3.3) ┤ 0
[3.3, 3.6) ┤▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 47 [3.3, 3.6) ┤ 0
[3.5, 3.8) ┤▇▇ 2 [3.5, 3.8) ┤▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 46
[3.8, 4.1) ┤▇ 1 [3.8, 4.1) ┤▇▇▇ 4
[4.0, 4.3) ┤ 0 [4.0, 4.3) ┤ 0
[4.3, 4.6) ┤ 0 [4.3, 4.6) ┤▇ 1
└ ┘ └ ┘
# of runs in range # of runs in range
```
To sum this up, the "new" version of this code (using real classes instead of structs) is 10% faster across 50 runs with a statistical significance confidence level of 99%. Histograms are for visual checksum.
2021-12-02 15:55:42 +09:00
d896746d69
Keep the generated source files when clean [Bug #18363 ]
2021-11-25 19:16:39 +09:00
ac152b3cac
Update dependencies
2021-11-21 16:21:18 +09:00
5c167a9778
ruby tool/update-deps --fix
2021-10-05 14:18:23 +09:00
6413dc27dc
dependency updates
2021-04-13 14:30:21 +09:00
5de38c41ae
ripper: fix a bug of Ripper::Lexer with syntax error and heredoc [Bug #17644 ]
2021-02-19 16:40:29 +09:00
b84b253a69
Fix Ripper with heredoc.
2021-01-17 12:58:13 +09:00
433a3be86a
ripper: call #pretty_print on also `state`
2021-01-04 23:37:00 +09:00
e33eb09b76
ripper: fix `#tok` on some error events [Bug 17345]
...
sorting alias target by event arity, and setup suitable `Elem` for error.
2020-12-19 17:32:39 +09:00
e0bdd54348
Ripper: Refined error callbacks [Bug #17345 ]
2020-12-15 21:36:23 +09:00
7898f4243f
ripper: return pushed new token instead of the token list
2020-12-15 10:26:50 +09:00
f5ca3ff4db
Store all kinds of syntax errors [Bug #17345 ]
2020-11-26 20:14:34 +09:00
4f5d14eb8c
[DOC] Ripper.{lex,tokenize} now always return full tokens. [ci skip]
2020-11-20 15:46:17 -08:00
69d871eeeb
[Feature #17276 ] Moved raise_errors support to Ripper::Lexer#parse
2020-11-20 17:18:27 +09:00
1800f3fa5c
Ripper.{lex,tokenize} return full tokens even if syntax error
...
yet another implements [Feature #17276 ]
2020-11-20 11:44:57 +09:00
1301bd8ca9
Update documentation for Ripper.{lex,tokenize,sexp,sexp_raw} [ci skip]
2020-11-17 21:26:56 -08:00
cd0877a93e
Support raise_errors keyword for Ripper.{lex,tokenize,sexp,sexp_raw}
...
Implements [Feature #17276 ]
2020-11-17 21:15:50 -08:00
5e3259ea74
fix public interface
...
To make some kind of Ractor related extensions, some functions
should be exposed.
* include/ruby/thread_native.h
* rb_native_mutex_*
* rb_native_cond_*
* include/ruby/ractor.h
* RB_OBJ_SHAREABLE_P(obj)
* rb_ractor_shareable_p(obj)
* rb_ractor_std*()
* rb_cRactor
and rm ractor_pub.h
and rename srcdir/ractor.h to srcdir/ractor_core.h
(to avoid conflict with include/ruby/ractor.h)
2020-11-18 03:52:41 +09:00
79df14c04b
Introduce Ractor mechanism for parallel execution
...
This commit introduces Ractor mechanism to run Ruby program in
parallel. See doc/ractor.md for more details about Ractor.
See ticket [Feature #17100 ] to see the implementation details
and discussions.
[Feature #17100 ]
This commit does not complete the implementation. You can find
many bugs on using Ractor. Also the specification will be changed
so that this feature is experimental. You will see a warning when
you make the first Ractor with `Ractor.new`.
I hope this feature can help programmers from thread-safety issues.
2020-09-03 21:11:06 +09:00
490010084e
sed -i '/rmodule.h/d'
2020-08-27 16:42:06 +09:00
756403d775
sed -i '/r_cast.h/d'
2020-08-27 15:03:36 +09:00
0da2a3f1fc
sed -i '\,2/extern.h,d'
2020-08-27 14:07:49 +09:00
d32e2bb02d
Allow references to $$ in Ripper DSL
2020-05-29 09:41:27 +09:00
9e41a75255
sed -i 's|ruby/impl|ruby/internal|'
...
To fix build failures.
2020-05-11 09:24:08 +09:00
d7f4d732c1
sed -i s|ruby/3|ruby/impl|g
...
This shall fix compile errors.
2020-05-11 09:24:08 +09:00
b7e1eda932
Suppress warnings by gcc 10.1.0-RC-20200430
...
* Folding results should not be empty.
If `OnigCodePointCount(to->n)` were 0, `for` loop using `fn`
wouldn't execute and `ncs` elements are not initialized.
```
enc/unicode.c:557:21: warning: 'ncs[0]' may be used uninitialized in this function [-Wmaybe-uninitialized]
557 | for (i = 0; i < ncs[0]; i++) {
| ~~~^~~
```
* Cast to `enum yytokentype`
Additional enums for scanner events by ripper are not included
in `yytokentype`.
```
ripper.y:7274:28: warning: implicit conversion from 'enum <anonymous>' to 'enum yytokentype' [-Wenum-conversion]
```
2020-05-04 12:28:24 +09:00
9e6e39c351
Merge pull request #2991 from shyouhei/ruby.h
...
Split ruby.h
2020-04-08 13:28:13 +09:00
05485868cb
Workaround for bison provided by scoop on mswin environment
2020-02-15 21:20:25 +09:00
9f99760daf
Get rid of use of special variables
...
Use `"\n"` and `IO#fileno` instead of `$/` and `$.` respectively.
[Feature #14240 ]
2020-01-20 16:58:58 +09:00
4e6bcac23e
Update dependencies in makefiles again
...
patch from https://travis-ci.org/ruby/ruby/jobs/638231960
2020-01-17 11:19:01 +09:00
179e402d8a
Updated dependencies on internal/warnings.h
...
Needed for `UNALIGNED_MEMBER_ACCESS` using `COMPILER_WARNING_`*
macros.
2019-12-31 11:14:19 +09:00
0c2d731ef2
update dependencies
2019-12-26 20:45:12 +09:00
265b5382b2
Allow `$10` and more in the Ripper DSL
2019-11-26 21:57:00 +09:00
e1b2341488
Update dependencies
2019-11-18 23:16:22 +09:00
ae33f93836
Update comment of Ripper.lex
...
This is follow up of 1f7cb4bee9
2019-11-13 09:50:58 +09:00
fb6a489af2
Revert "Method reference operator"
...
This reverts commit 67c5747369#16275 ]
2019-11-12 17:24:48 +09:00
f48655d04d
Remove unneeded exec bits from some files
...
I noticed that some files in rubygems were executable, and I could think
of no reason why they should be.
In general, I think ruby files should never have the executable bit set
unless they include a shebang, so I run the following command over the
whole repo:
```bash
find . -name '*.rb' -type f -executable -exec bash -c 'grep -L "^#!" $1 || chmod -x $1' _ {} \;
```
2019-11-09 21:36:30 +09:00
d62abc47c8
Suppress unused variable warning
2019-11-07 23:33:45 +09:00
215e4b8721
* expand tabs. [ci skip]
...
Tabs were expanded because previously the file did not have any tab indentation.
Please update your editor config, and use misc/expand_tabs.rb in the pre-commit hook.
2019-10-21 11:50:44 +09:00
44da5a1c87
Look up event ID offsets by token as index
2019-10-21 11:45:54 +09:00
ca7de0334d
Also BDOT2 and BDOT3 are operators
2019-10-20 20:45:56 +09:00
1d435bd51a
Extend Ripper DSL to set to pseudo variables other than `$$`
2019-10-19 17:19:27 +09:00
ea68bb914a
Changed numbered parameter prefix
2019-09-24 21:57:54 +09:00
f223ab47e6
[DOC] Update output of Ripper.sexp [ci skip]
2019-09-07 13:50:05 +09:00
2ed68d0ff9
Revert "Add pipeline operator [Feature #15799 ]"
...
This reverts commits:
* d365fd5a02d780c36624aa7211836b043f010c28043f010c28f169043d81http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-core/94645
2019-08-29 15:27:59 +09:00
8877dbe400
ext/ripper/lib/ripper/lexer.rb: Consistently use `Array#push`
...
instead of <<. All the other callsites use `push`.
2019-08-07 03:13:17 +09:00
Yusuke Endoh
Nobuyoshi Nakada
schneems
卜部昌平
Shugo Maeda
manga_osyo
Nobuhiro IMAI
Jeremy Evans
Koichi Sasada
Hiroshi SHIBATA
Kazuhiro NISHIYAMA
Yuichiro Kaneko
David Rodríguez
git