Adds the `syntax_suggest` syntax error display tool to Ruby through the same mechanism as `error_highlight` and `did_you_mean`. Reference ticket: https://bugs.ruby-lang.org/issues/18159close#4845
## What is syntax_suggest?
When a syntax error is raised by requiring a file, dead_end will use a combination of indentation and lexing to identify the problem.
> Note: Previously this tool was named `dead_end`.
## Known issues
- SyntaxSearch's approach of showing syntax errors only works through integration with `require`, `load`, `autoload`, and `require_relative` (since it monkeypatches them to detect syntax errors). It does not work with direct Ruby file invocations https://github.com/zombocom/dead_end/issues/31.
- This causes failure in the test suite (test_expected_backtrace_location_when_inheriting_from_basic_object_and_including_kernel) and confusion when inspecting backtraces if there's a different error when trying to require a file such as measuring memory (https://github.com/zombocom/syntax_suggest/issues/124#issuecomment-1006705016).
- Discussed fix. We previously talked about opening up `SyntaxError` to be monkeypatched in the same way that other gems hook into `NoMethodError`. This is currently not possible and requires development work. When we last talked about it at RubyKaigi Nobu expressed an ability to make such a change.
* Simplify around `USE_YJIT` macro
- Use `USE_YJIT` macro only instead of `YJIT_BUILD`.
- An intermediate macro `YJIT_SUPPORTED_P` is no longer used.
* Bail out if YJIT is enabled on unsupported platforms
rb_ary_tmp_new suggests that the array is temporary in some way, but
that's not true, it just creates an array that's hidden and not on the
transient heap. This commit renames it to rb_ary_hidden_new.
YJIT is now a build-time opt-in so on platforms that YJIT could support
it could still be unavailable due to user discretion. Use MJIT for --jit
and don't display YJIT related command line options in --help when YJIT
is not included in the build.
Since enabling YJIT or MJIT drastically changes what could go wrong at
runtime, it's good to be front and center about whether they are enabled
when dumping a crash report. Previously, `RUBY_DESCRIPTION` and the
description printed when crashing can be different when a JIT is on.
Introduce a new internal data global, `rb_dynamic_description`, and set
it to be the same as `RUBY_DESCRIPTION` during initialization; use it
when crashing.
* version.c: Init_ruby_description(): Initialize and use
`rb_dynamic_description`.
* error.c: Change crash reports to use `rb_dynamic_description`.
* ruby.c: Call `Init_ruby_description()` earlier. Slightly more work
for when we exit right after printing the description but that
was deemed acceptable.
* include/ruby/version.h: Talk about how JIT info is not in
`ruby_description`.
* test/-ext-/bug_reporter/test_bug_reporter.rb: Remove handling for
crash description being different from `RUBY_DESCRIPTION`.
* test/ruby/test_rubyoptions.rb: ditto
Co-authored-by: Nobuyoshi Nakada <nobu@ruby-lang.org>
Co-authored-by: Alan Wu <alanwu@ruby-lang.org>
In December 2021, we opened an [issue] to solicit feedback regarding the
porting of the YJIT codebase from C99 to Rust. There were some
reservations, but this project was given the go ahead by Ruby core
developers and Matz. Since then, we have successfully completed the port
of YJIT to Rust.
The new Rust version of YJIT has reached parity with the C version, in
that it passes all the CRuby tests, is able to run all of the YJIT
benchmarks, and performs similarly to the C version (because it works
the same way and largely generates the same machine code). We've even
incorporated some design improvements, such as a more fine-grained
constant invalidation mechanism which we expect will make a big
difference in Ruby on Rails applications.
Because we want to be careful, YJIT is guarded behind a configure
option:
```shell
./configure --enable-yjit # Build YJIT in release mode
./configure --enable-yjit=dev # Build YJIT in dev/debug mode
```
By default, YJIT does not get compiled and cargo/rustc is not required.
If YJIT is built in dev mode, then `cargo` is used to fetch development
dependencies, but when building in release, `cargo` is not required,
only `rustc`. At the moment YJIT requires Rust 1.60.0 or newer.
The YJIT command-line options remain mostly unchanged, and more details
about the build process are documented in `doc/yjit/yjit.md`.
The CI tests have been updated and do not take any more resources than
before.
The development history of the Rust port is available at the following
commit for interested parties:
1fd9573d8b
Our hope is that Rust YJIT will be compiled and included as a part of
system packages and compiled binaries of the Ruby 3.2 release. We do not
anticipate any major problems as Rust is well supported on every
platform which YJIT supports, but to make sure that this process works
smoothly, we would like to reach out to those who take care of building
systems packages before the 3.2 release is shipped and resolve any
issues that may come up.
[issue]: https://bugs.ruby-lang.org/issues/18481
Co-authored-by: Maxime Chevalier-Boisvert <maximechevalierb@gmail.com>
Co-authored-by: Noah Gibbs <the.codefolio.guy@gmail.com>
Co-authored-by: Kevin Newton <kddnewton@gmail.com>
Add a hook point to initialize extra extension libraries. The default
hook function is replaced when linking a strong `Init_extra_exts`
symbol. A builder can insert an object file that defines Init_extra_exts
by XLDFLAGS.
```
compiling ../ruby.c
../ruby.c:1547:17: error: implicit declaration of function 'setup_yjit_options' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
setup_yjit_options(s, &opt->yjit);
^
../ruby.c:1547:17: note: did you mean 'setup_mjit_options'?
../ruby.c:1122:1: note: 'setup_mjit_options' declared here
setup_mjit_options(const char *s, struct mjit_options *mjit_opt)
^
../ruby.c:1547:45: error: no member named 'yjit' in 'struct ruby_cmdline_options'; did you mean 'mjit'?
setup_yjit_options(s, &opt->yjit);
^~~~
mjit
../ruby.c:192:25: note: 'mjit' declared here
struct mjit_options mjit;
^
../ruby.c:1924:28: error: no member named 'yjit' in 'struct ruby_cmdline_options'; did you mean 'mjit'?
rb_yjit_init(&opt->yjit);
^~~~
mjit
../ruby.c:192:25: note: 'mjit' declared here
struct mjit_options mjit;
^
3 errors generated.
```
* Rename --jit to --mjit
[Feature #18349]
* Fix a few more --jit references
* Fix MJIT Actions
* More s/jit/mjit/ and re-introduce --disable-jit
* Update NEWS.md
* Fix test_bug_reporter_add
* Add --yjit-no-type-prop so we can test YJIT without type propagation
* Fix typo in command line option
* Leave just two test workflows enable for YJIT
TestRubyOptions#test_enable was broken on OpenBSD after the yjit
merge. --yjit (and --enable-all, which enables --yjit) fails on
OpenBSD because yjit uses an insecure mmap call (both writable
and executable), in alloc_exec_mem, which OpenBSD does not allow.
This can probably be reverted if yjit switches to a more secure
mmap design (writable xor executable). This would involve
initially calling mmap with PROT_READ | PROT_WRITE, and after writing
of executable code has finished, using mprotect to switch to
PROT_READ | PROT_EXEC. I believe Firefox uses this approach for
their Javascript engine since Firefox 46.
Previously, options such as "--yjit123" would enable YJIT. Additionally,
the error message for argument parsing mentioned "--jit-..." instead of
"--yjit-...".
Must not be a bad idea to improve documents. [ci skip]
In fact many functions declared in the header file are already
documented more or less. They were just copy & pasted, with applying
some style updates.
When runtime_libruby_path does not include '/', it attempts to call
rb_str_resize with negative length. This change makes sure that the
length non-negative.
Co-Authored-By: xtkoba (Tee KOBAYASHI) <xtkoba+ruby@gmail.com>
Also document that both :deprecated and :experimental are supported
:category option values.
The locations where warnings were marked as deprecation warnings
was previously reviewed by shyouhei.
Comment a couple locations where deprecation warnings should probably
be used but are not currently used because deprecation warning
enablement has not occurred at the time they are called
(RUBY_FREE_MIN, RUBY_HEAP_MIN_SLOTS, -K).
Add assert_deprecated_warn to test assertions. Use this to simplify
some tests, and fix failing tests after marking some warnings with
deprecated category.
This changes the behavior, which I'm not sure is acceptable.
However, it's odd to allow an option to be combined, but change
the behavior of the option when combined.
* Use UTF-8 as default for Encoding.default_external on Windows
* Document UTF-8 change on Windows to Encoding.default_external
fix https://bugs.ruby-lang.org/issues/16604
As fork(2) is deprecated, its calls must be guarded by
`COMPILER_WARNING_IGNORED(-Wdeprecated-declarations)`.
All usages of fork(2) in process have been alread guarded. A new call
to fork(2) was added in ruby.c with f22c4ff359.
This caused a build failure on Solaris 11.
It may hide a bug to guard big code unnecessarily, so this change
introduces a simple wrapper "rb_fork" whose definition is guarded, and
replaces all calls to fork(2) with the wrapper function.
Saves comitters' daily life by avoid #include-ing everything from
internal.h to make each file do so instead. This would significantly
speed up incremental builds.
We take the following inclusion order in this changeset:
1. "ruby/config.h", where _GNU_SOURCE is defined (must be the very
first thing among everything).
2. RUBY_EXTCONF_H if any.
3. Standard C headers, sorted alphabetically.
4. Other system headers, maybe guarded by #ifdef
5. Everything else, sorted alphabetically.
Exceptions are those win32-related headers, which tend not be self-
containing (headers have inclusion order dependencies).
This is a secret feature for me. It's only for testing and any behavior
with this flag override is unsupported.
I needed this because I sometimes want to add debug options but do not
want to disable optimizations, for using Linux perf.
This removes the related tests, and puts the related specs behind
version guards. This affects all code in lib, including some
libraries that may want to support older versions of Ruby.
This removes the security features added by $SAFE = 1, and warns for access
or modification of $SAFE from Ruby-level, as well as warning when calling
all public C functions related to $SAFE.
This modifies some internal functions that took a safe level argument
to no longer take the argument.
rb_require_safe now warns, rb_require_string has been added as a
version that takes a VALUE and does not warn.
One public C function that still takes a safe level argument and that
this doesn't warn for is rb_eval_cmd. We may want to consider
adding an alternative method that does not take a safe level argument,
and warn for rb_eval_cmd.
The relation between parser_param#base_block and #in_main were very
subtle.
A main script (that is passed via a command line) was parsed under
base_block = TOPLEVEL_BINDING and in_main = 1.
A script loaded by Kernel#require was parsed under
base_block = NULL and in_main = 0.
If base_block is non-NULL and in_main == 0, it is parsed by Kernel#eval
or family.
However, we know that TOPLEVEL_BINDING has no local variables when a
main script is parsed. So, we don't have to parse a main script under
base_block = TOPLEVEL_BINDING.
Instead, this change parses a main script under base_block = 0.
If base_block is non-NULL, it is parsed by Kernel#eval or family.
By this simplication, "in_main" is no longer needed.
We can check the function pointer passed to rb_define_global_function
like we do so in rb_define_method. It turns out that almost anybody
is misunderstanding the API.
After 5e86b005c0, I now think ANYARGS is
dangerous and should be extinct. This commit uses rb_gvar_getter_t /
rb_gvar_setter_t for rb_define_hooked_variable /
rb_define_virtual_variable which revealed lots of function prototype
inconsistencies. Some of them were literally decades old, going back
to dda5dc00cf.
Terminate the input from a TTY by 2 ^D at the middle of line, like
as many programs, `cat`, `perl` and so on, do. By the first ^D,
the line will be sent without a newline, and then EOF will be send
by the next ^D.
* ruby.c (process_options): feature options in command line
arguments take precedence over options in RUBYOPT environment
variable. [ruby-core:92052] [Bug #15738]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67388 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* --jit-min-calls: 5 -> 10000
--jit-min-calls=5 obviously can compile non hotspot. This was not a
problem for MJIT-benchmarks and Optcarrot because the former has very
few hot optimiziable methods and the latter is likely to trigger
compilation of hotspot by its intensive calls to optimizable hotspot
methods and has a very short window to allow limited compilations.
In real-world applications, it has more time to compile more methods and
it pressures computer's limited resources like icache. We should avoid
compiling too many methods. Also compiling many methods exhausts time
budget for compilation in one ruby process lifetime and delays the "JIT
compaction" of Ruby 2.6.
JVM is known to use 1,500 for C1 (client) compiler and 10,000 for C2
(server) compiler for -XX:CompileThreshold by default.
https://docs.oracle.com/javase/8/embedded/develop-apps-platforms/codecache.htm
When things are called X,000 times, requiring 10,000 can eliminate
compilation of methods which are called only once in these X,000
iterations and obviously not hotspot. And in fact things like
unicorn-worker-killer restarts unicorn process every 4096 requests.
So I felt 10,000 is good for such an environment.
* --jit-max-cache: 1000 -> 100
By the same reason stated above, we should not allow compiling many
methods especially on MJIT which has a larger overhead due to poor code
locality by dlopen and whose code is also likely to be bigger by just
inlining many VM instructions with -O3 rather than directly generating
low-level code.
In JVM -XX:ReservedCodeCacheSize is 32M for reserved and 48M for maximum.
--jit-max-cache=1,000 could be closer to it, but in this case MJIT's
compilation is slow due to data synchronization between threads (to be
improved in Ruby 2.7 though) and we do not want to delay the "JIT
compaction" for a long time.
So I chose a really conservative number for this, but by having method
inlining in the future, wider range could be optimized even with this
value.
* Optcarrot
--disable-gems, --benchmark Lan_Master.nes 12 attempts.
No significant impact.
| r67276 | r67276 --jit | after --jit |
|:-------------------|:------------------|:------------------|
| 50.44369263063978 | 72.87390680773056 | 73.47873485047297 |
| 50.58788746124193 | 78.06820808947026 | 78.29723420171945 |
| 50.77509250801378 | 80.29010348842613 | 78.94689404460769 |
| 50.935361702064405 | 80.42796829926374 | 80.39539527351525 |
| 51.27352672981195 | 81.98758158033202 | 81.6754198664817 |
| 51.720715743242124 | 82.00118535811626 | 82.22960569251283 |
| 51.89643169822524 | 82.2290091613556 | 82.5013636146388 |
| 51.95895898113868 | 82.37318990939565 | 82.74002377794454 |
| 52.10124886807968 | 82.48796686037502 | 83.23354941183932 |
| 52.292280637519376 | 83.0265226541066 | 84.01552618012572 |
| 52.51856237784144 | 83.8797360318052 | 84.8588319093393 |
| 52.65076845986818 | 84.80037351256634 | 85.13577756273656 |
* Railsbench
`WARMUP=20000 BENCHMARK=1000 bin/bench` of https://github.com/k0kubun/railsbench.
It gets closer to --disable=jit.
| | r67276 | r67276 | after |
| | | --jit | --jit |
|:----------|:-------|:-------|:-------|
| req/s | 891.3 | 742.2 | 841.5 |
|:----------|:-------|:-------|:-------|
| 50%ile ms | 1.00 | 1.21 | 1.08 |
| 66%ile ms | 1.02 | 1.24 | 1.09 |
| 75%ile ms | 1.03 | 1.28 | 1.10 |
| 80%ile ms | 1.03 | 1.30 | 1.11 |
| 90%ile ms | 1.09 | 1.42 | 1.15 |
| 95%ile ms | 1.32 | 1.65 | 1.27 |
| 98%ile ms | 4.79 | 2.23 | 1.81 |
| 99%ile ms | 5.68 | 7.52 | 6.64 |
|100%ile ms | 6.52 | 9.69 | 8.59 |
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67277 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* eval_error.c (print_errinfo): defer escaping control char in
error messages until writing to stderr, instead of quoting at
building the message. [ruby-core:90853] [Bug #15497]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66753 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* ruby.c (process_options): script_compiled events are missed on
command line -e or specified file. this commit fix it.
[Bug #15471]
This patch should be backport to Ruby 2.6 branch.
* vm_core.h (rb_exec_event_hook_script_compiled): introduce utility
function to invoke a script_compiled event.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66595 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
In some places, both JIT and MJIT are being used, but it could be
confusing for new comers. We're not explaining MJIT on NEWS file or release
notes as well. So we consider MJIT as an internal term of implementation
like YARV.
configure.ac: ditto
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65810 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* configure.ac: introduce new configure option `--enable-mjit` and
`--disable-mjit`. Default is "enable".
`--disable-mjit` disables all of MJIT features so that `ruby --jit`
can't enable MJIT.
This option affect a macro `USE_MJIT`.
This change remove `--enable/disable-install-mjit-header` option.
* Makefile.in: introduce the `ENABLE_MJIT` variable.
* common.mk: use `ENABLE_MJIT` option.
* internal.h: respect `USE_MJIT`. Same as other *.c, *.h.
* test/ruby/test_jit.rb: check `ENABLE_MJIT` key of rbconfg.rb.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65204 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* version.c (Init_ruby_description): separate to initialize
RUBY_DESCRIPTION constant according to mjit.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64264 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* ruby.c (external_str_new_cstr): strings come from the external
should be tainted. [ruby-dev:50596] [Bug #14941]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64071 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
by promoting jit to feature flag.
mjit.h: update comment about mjit_opts.on
test_rubyoptions.rb: add test for switching JIT enablement
"--jit" flag usage may be deprecated later, but not discussed yet.
[Feature #14878]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63995 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* ruby.c (process_options): as DidYouMean requires Rubygems, disable
the former when the latter is disabled too.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63977 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* ruby.c (cmdline_options_init): set up mjit.on flag by
MJIT_FORCE_ENABLE in the initialization function.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63822 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* ruby.c (ruby_init_loadpath_safe): moved libdir to the block
where it is used.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63477 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* ruby.c (runtime_libruby_path): hoisted out platform dependent
routine to get the loaded runtime library path.
cygwin_conv_path does path separator and WCHAR to UTF-8
conversions too.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63476 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
We need to ensure aligned memory access by allocating
another memory region.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63388 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* ruby.c (ruby_init_loadpath_safe): fix compilation error when
ENABLE_MULTIARCH but not universal binary.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63270 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* ruby.c (ruby_init_loadpath_safe): store prefix and archlibdir
paths.
* mjit.c (compile_c_to_so, init_header_filename): use just one
library path on Windows.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63269 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* ruby.c (usage): fix the description of `--verbose` option, which
does not print the version number unlike `-v` option.
[ruby-core:86307] [Bug #14633]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62927 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
in version output.
version.h: ditto
ruby.c: propagate option for it
common.mk: updated dependency for version.c
mjit.c: overwrites the RUBY_DESCRIPTION to have +JIT when --jit is passed
test/ruby/test_rubyoptions.rb: add test for them
Only `ruby --jit -v` will have "+JIT", but this is intentional.
This may not be convenient for debugging by ticket with `ruby -v`,
but it's convenient for benchmark tools that pass options (--jit)
when showing it. At least such behavior is planned for benchmark_driver.gem
and this behavior is designed for it. Other benchmark tools are
recommended to follow the behavior too if they show version.
RUBY_DESCRIPTION might be useful for it too.
The position of "+JIT" is changed from original proposal because other
platforms like JRuby and TruffleRuby end it with archtecture.
It's made similar to JRuby, but it's upper-cased because Matz made approval
for "+JIT" in the ticket.
Example:
$ ruby -v
ruby 2.6.0dev (2018-02-22 trunk 62529) [x86_64-linux]
$ ruby --jit -v
ruby 2.6.0dev (2018-02-22 trunk 62529) +JIT [x86_64-linux]
After --jit is made default in the future, this output may be removed.
So do not rely on this output if possible.
[Feature #14462]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62530 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* ruby.c (setup_mjit_options): removed --jit-cc option, since
mjit header is affected by generated config.h which depends on
the given compiler, so it cannot work with different compilers.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62379 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* internal.h (ruby_engine): remove declaration of ruby_engine because
it's declared at ruby/version.h.
* ruby.c: include ruby/version.h for ruby_engine.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62274 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
so that MJIT can work if Ruby is distributed as prebuilt binary.
Now mjit_init() depends on the internal const TMP_RUBY_PREFIX which is
only available after ruby_init_loadpath_safe() (L1608) and before
ruby_init_prelude() (L1681). So the place of mjit_init() is moved.
Makefile.in: Removed static prefix from MJIT_HEADER_ISNTALL_DIR macro.
And this removes the unused LIBRUBY_LIBDIR macro as well.
win32/Makefile.sub: ditto.
Patch by: Lars Kanis <lars@greiz-reinsdorf.de>
[Bug #14445]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62238 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
that allows to JIT-compile Ruby methods by generating C code and
using C compiler. See the first comment of mjit.c to know what this
file does.
mjit.c is authored by Vladimir Makarov <vmakarov@redhat.com>.
After he invented great method JIT infrastructure for MRI as MJIT,
Lars Kanis <lars@greiz-reinsdorf.de> sent the patch to support MinGW
in MJIT. In addition to merging it, I ported pthread to Windows native
threads. Now this MJIT infrastructure can be compiled on Visual Studio.
This commit simplifies mjit.c to decrease code at initial merge. For
example, this commit does not provide multiple JIT threads support.
We can resurrect them later if we really want them, but I wanted to minimize
diff to make it easier to review this patch.
`/tmp/_mjitXXX` file is renamed to `/tmp/_ruby_mjitXXX` because non-Ruby
developers may not know the name "mjit" and the file name should make
sure it's from Ruby and not from some harmful programs. TODO: it may be
better to store this to some temporary directory which Ruby is already using
by Tempfile, if it's not bad for performance.
mjit.h: New. It has `mjit_exec` interface similar to `vm_exec`, which is
for triggering MJIT. This drops interface for AOT compared to the original
MJIT.
Makefile.in: define macros to let MJIT know the path of MJIT header.
Probably we can refactor this to reduce the number of macros (TODO).
win32/Makefile.sub: ditto.
common.mk: compile mjit.o and mjit_compile.o. Unlike original MJIT, this
commit separates MJIT infrastructure and JIT compiler code as independent
object files. As initial patch is NOT going to have ultra-fast JIT compiler,
it's likely to replace JIT compiler, e.g. original MJIT's compiler or some
future JIT impelementations which are not public now.
inits.c: define MJIT module. This is added because `MJIT.enabled?` was
necessary for testing.
test/lib/zombie_hunter.rb: skip if `MJIT.enabled?`. Obviously this
wouldn't work with current code when JIT is enabled.
test/ruby/test_io.rb: skip this too. This would make no sense with MJIT.
ruby.c: define MJIT CLI options. As major difference from original MJIT,
"-j:l"/"--jit:llvm" are renamed to "--jit-cc" because I want to support
not only gcc/clang but also cl.exe (Visual Studio) in the future. But it
takes only "--jit-cc=gcc", "--jit-cc=clang" for now. And only long "--jit"
options are allowed since some Ruby committers preferred it at Ruby
developers Meeting on January, and some of options are renamed.
This file also triggers to initialize MJIT thread and variables.
eval.c: finalize MJIT worker thread and variables.
test/ruby/test_rubyoptions.rb: fix number of CLI options for --jit.
thread_pthread.c: change for pthread abstraction in MJIT. Prefix rb_ for
functions which are used by other files.
thread_win32.c: ditto, for Windows. Those pthread porting is one of major
works that YARV-MJIT created, which is my fork of MJIT, in Feature 14235.
thread.c: follow rb_ prefix changes
vm.c: trigger MJIT call on VM invocation. Also trigger `mjit_mark` to avoid
SEGV by race between JIT and GC of ISeq. The improvement was provided by
wanabe <s.wanabe@gmail.com>.
In JIT compiler I created and am going to add in my next commit, I found
that having `mjit_exec` after `vm_loop_start:` is harmful because the
JIT-ed function doesn't proceed other ISeqs on RESTORE_REGS of leave insn.
Executing non-FINISH frame is unexpected for my JIT compiler and
`exception_handler` triggers executions of such ISeqs. So `mjit_exec`
here should be executed only when it directly comes from `vm_exec` call.
`RubyVM::MJIT` module and `.enabled?` method is added so that we can skip
some tests which don't expect JIT threads or compiler file descriptors.
vm_insnhelper.h: trigger MJIT on method calls during VM execution.
vm_core.h: add fields required for mjit.c. `bp` must be `cfp[6]` because
rb_control_frame_struct is likely to be casted to another struct. The
last position is the safest place to add the new field.
vm_insnhelper.c: save initial value of cfp->ep as cfp->bp. This is an
optimization which are done in both MJIT and YARV-MJIT. So this change
is added in this commit. Calculating bp from ep is a little heavy work,
so bp is kind of cache for it.
iseq.c: notify ISeq GC to MJIT. We should know which iseq in MJIT queue
is GCed to avoid SEGV. TODO: unload some GCed units in some safe way.
gc.c: add hooks so that MJIT can wait GC, and vice versa. Simultaneous
JIT and GC executions may cause SEGV and so we should synchronize them.
cont.c: save continuation information in MJIT worker. As MJIT shouldn't
unload JIT-ed code which is being used, MJIT wants to know full list of
saved execution contexts for continuation and detect ISeqs in use.
mjit_compile.c: added empty JIT compiler so that you can reuse this commit
to build your own JIT compiler. This commit tries to compile ISeqs but
all of them are considered as not supported in this commit. So you can't
use JIT compiler in this commit yet while we added --jit option now.
Patch author: Vladimir Makarov <vmakarov@redhat.com>.
Contributors:
Takashi Kokubun <takashikkbn@gmail.com>.
wanabe <s.wanabe@gmail.com>.
Lars Kanis <lars@greiz-reinsdorf.de>.
Part of Feature 12589 and 14235.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62189 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* ruby.c (ruby_init_loadpath_safe): removed code using fixed size
path buffer. relative load path is supported only on platforms
where dladdr is available, or on Windows, so !VARIABLE_LIBPATH
code is not used.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62086 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* ruby.c (process_options): fallback to the encoding of the script
name since rb_realpath_internal() cannot convert the encoding
when it is ASCII-8BIT.
* test/ruby/test_rubyoptions.rb (test___dir__encoding): explicitly
pass environment variables for locale as they are overriden by
invoke_ruby.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61297 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
This re-introduces r60485.
This reverts commit 5a176b75b1.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@60488 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
NODEs in AST are no longer objects managed by GC. This change will
remove the restriction imposed by the GC. For example, a NODE can use
more than five words (this is my primary purpose; we want to store the
position data for each NODE, for coverage library), or even a NODE can
have variable length (some kinds of NODEs have unused fields).
To do this, however, we need more work, since Ripper still uses T_NODE
objects managed by the GC.
The life time of NODEs is more obvious than other kinds of objects; they
are created at parsing, and they become disused immediately after
compilation. This change releases all NODEs by a few `xfree`s after
compilation, so performance will be improved a bit. In extreme example,
`eval("x=1;" * 10000000)` runs much faster (40 sec. -> 7.8 sec. on my
machine).
The most important part of this change is `ast_t` struct, which has
three contents: (1) NODE buffer (malloc'ed memory), (2) a reference to
the root NODE, and (3) an array that contains objects that must be
marked during parsing (such as literal objects). Some functions that
had received `NODE*` arguments, must now receive `ast_t*`.
* node.c, node.h: defines `ast_t` struct and related operations.
* gc.c, internal.h: defines `imemo_ast`.
* parse.y: makes `parser_params` struct have a reference to `ast_t`.
Instead of `rb_node_newnode`, use `rb_ast_newnode` to create a NODE.
* iseq.c, load.c, ruby.c, template/prelude.c.tmpl: modifies some
functions to handle `ast_t*` instead of `NODE*`.
* test/ruby/test_gc.rb: ad-hoc fix for a failed test. The test assumes
GC eden is increased at startup by NODE object creation. However,
this change now create no NODE object, so GC eden is not necessarily
increased.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@60485 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* ruby.c (process_options): convert the real path of the script to locale
encoding if its encoding is not locale (maybe UTF-8) on Windows/OS X.
this change makes the encoding of __dir__ to the same encoding of __FILE__
when the script name is passed from commandline.
* test/ruby/test_options.rb (test___dir__encoding): test for this change.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@60300 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* ruby.c (ruby_setproctitle): raise if the argument contains NUL
char. process title is a NUL-terminated string.
[ruby-core:82425] [Bug #13829]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@60215 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* ruby.c (proc_options): set to paragraph mode, if -00 is given,
as well as perl and -R0 option in 0.49.
[ruby-core:81987] [Bug #13736]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@59970 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* ruby.c (moreswitches): process all words as options only in
an environment variable, but not in a shebang line.
[ruby-core:82267] [Bug #13786]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@59564 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* ruby.c (dladdr_path, ruby_set_argv): add guards for origarg.
[ruby-core:82272] [Bug #13788]
* ruby.c (proc_options, process_options, ruby_process_options):
set origarg if not set yet.
* ruby.c (process_options): prefer argv in the argument to origarg
as program name.
* ruby.c (ruby_sysinit): set origarg only if argc and argv seem
valid.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@59549 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* configure.in: define RUBY_DEVEL only in the trunk.
* gc.c: enable runtime rgengc debug if RUBY_DEVEL
* ruby.c (debug_option): enable RUBY_DEBUG in --debug option only
if RUBY_DEVEL.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@59131 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* ruby.c (debug_option): parse options in --debug command line
option same as RUBY_DEBUG env. available only in the trunk.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@59129 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* ruby.c (process_options): keep script name in UTF-8 if UTF8_PATH
to get rid of loss by conversion.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@59041 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* ruby.c (dladdr_path): dladdr is provided on recent cygwin, but
GetModuleFileNameW is used instead of it.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@59007 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
On DOSish systems, there is a warning message for \r\n line endings on shebang line.
Improve this message from "shebang line ends with \r may cause a problem"
to "shebang line ending with \r may cause problems".
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58981 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* ruby.c (load_file): move opened file to an argument, to reduce
open/close calls in the near future.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58938 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* ruby.c (ruby_cmdline_options_t): reordered members and turned
simple flags into bit fields to reduce the size (136->104 on
LP64).
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58937 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* ruby.c (process_options): encode script name to locale encoding
instead of associate, if UTF-8 path.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58767 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* ruby.c (load_file_internal): define DATA here instead of ensure
func.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@57485 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* ruby.c (process_options): dump specified informations all, not
only first one.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@57017 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* ruby.c (open_load_file): retry after GC when the limit for open
file descriptors reached.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@56463 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* ruby.c (open_load_file): FILE_ALT_SEPARATOR and EXEEXT are
config.status variables, not available in config.h.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@56400 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* ruby.c (open_load_file): bind the open fd to an IO instance
before waiting FIFO, not to leak the fd if interrupted.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@56388 b2dd03c8-39d4-4d8f-98ff-823fe69b080e