Fixing 4bcd5981e8/mjit.c (L338)
should be the right solution for this. We may not be able to free the cc immediately.
Plus, we're not copying cc but just holding references to be marked. cc
should be GC-ed once jit_unit is freed.
jit_unit to avoid marking wrong cc entries when inlined iseq is compiled
multiple times, resolving the TODO added by daf7c48d88.
This obviates pseudo jit_unit in inlined iseq introduced by 7ec2359374
and fixes memory leak of the adhoc unit.
* Fix incorrect calls to `Gem.ensure_gem_subdirectories`
This method doesn't take keyword args.
* Remove stuff no longer necessary
Now `Gem.ensure_gem_subdirectories` is doing its job, so some stuff is
no longer needed.
* Use the proper method for default gems
* Respect DESTDIR when creating rubygems folder layout
* Use `Gem.default_specifications_dir`
There is a test directory named test/-ext-. Because this directry
starts with a hyphen, we have to cheat test/unit in order for it to
run the tests underneath. TESTS='-- -ext' worked for a long time.
Let's not break that maneuver.
MiniTest::Unit (superclass of Test::Unit::Runner) does not has
default seed parameter, but assigned after initializing.
However some tests use MiniTest::Unit without setup of seed option
and it cases unexpected test failures. To solve this issue, add
default seed parameter 42.
The local `path` variable does not provide any additional value and was
kept around just for clarity for easier review of the `extrac_files`
method move.
Extract bundled gems under ".bundle/gems" and get rid of
duplication which cause constant redefinition warnings at
`test-all` after `extract-gems` and `test-bundler`.
This patch contains several ideas:
(1) Disposable inline method cache (IMC) for race-free inline method cache
* Making call-cache (CC) as a RVALUE (GC target object) and allocate new
CC on cache miss.
* This technique allows race-free access from parallel processing
elements like RCU.
(2) Introduce per-Class method cache (pCMC)
* Instead of fixed-size global method cache (GMC), pCMC allows flexible
cache size.
* Caching CCs reduces CC allocation and allow sharing CC's fast-path
between same call-info (CI) call-sites.
(3) Invalidate an inline method cache by invalidating corresponding method
entries (MEs)
* Instead of using class serials, we set "invalidated" flag for method
entry itself to represent cache invalidation.
* Compare with using class serials, the impact of method modification
(add/overwrite/delete) is small.
* Updating class serials invalidate all method caches of the class and
sub-classes.
* Proposed approach only invalidate the method cache of only one ME.
See [Feature #16614] for more details.
Now, rb_call_info contains how to call the method with tuple of
(mid, orig_argc, flags, kwarg). Most of cases, kwarg == NULL and
mid+argc+flags only requires 64bits. So this patch packed
rb_call_info to VALUE (1 word) on such cases. If we can not
represent it in VALUE, then use imemo_callinfo which contains
conventional callinfo (rb_callinfo, renamed from rb_call_info).
iseq->body->ci_kw_size is removed because all of callinfo is VALUE
size (packed ci or a pointer to imemo_callinfo).
To access ci information, we need to use these functions:
vm_ci_mid(ci), _flag(ci), _argc(ci), _kwarg(ci).
struct rb_call_info_kw_arg is renamed to rb_callinfo_kwarg.
rb_funcallv_with_cc() and rb_method_basic_definition_p_with_cc()
is temporary removed because cd->ci should be marked.
Namely recent Sun C compiler has this function, and is not a GCC.
Meanwhile the code without RUBY_JMP_BUF assumes GCC. We have to define
the macro when we detect __builtin_setjmp for non-GCC compilers.
To output to the STDOUT of the parent process according to the
parallel test protocol, should send to the `MiniTest::Unit.output`
instead of each own STDOUT.
This fixes following error that sometimes happens once in a few times
on Travis s390x environment.
```
$ tool/travis_retry.sh sudo -E apt-add-repository -y "ppa:ubuntu-toolchain-r/test"
+ sudo -E apt-add-repository -y ppa:ubuntu-toolchain-r/test
Error: retrieving gpg key timed out.
```
Saves comitters' daily life by avoid #include-ing everything from
internal.h to make each file do so instead. This would significantly
speed up incremental builds.
We take the following inclusion order in this changeset:
1. "ruby/config.h", where _GNU_SOURCE is defined (must be the very
first thing among everything).
2. RUBY_EXTCONF_H if any.
3. Standard C headers, sorted alphabetically.
4. Other system headers, maybe guarded by #ifdef
5. Everything else, sorted alphabetically.
Exceptions are those win32-related headers, which tend not be self-
containing (headers have inclusion order dependencies).
Now, C functions written by __builtin_cexpr!(code) and others are
named as "__builtin_inline#{n}". However, it is difficult to know
what the function is. This patch rename them into
"__builtin_foo_#{lineno}" when cexpr! is in 'foo' method.
On cross-compilation, compiled binary can no be created because
compiled binary should be created by same interpreter (on cross-
compilation, host ruby is used to build ruby (BASERUBY)).
So that cross-compilation system loads required scripts in text.
It is same as miniruby.
prelude.c is an automatically generated file by template/prelude.c.tmpl.
However it does not contain any required functions. So remove it from
dependency.
Also miniprelude.c is included by mini_builtin.c and does not need
to make miniprelude.o.
This commit introduces an "inline ivar cache" struct. The reason we
need this is so compaction can differentiate from an ivar cache and a
regular inline cache. Regular inline caches contain references to
`VALUE` and ivar caches just contain references to the ivar index. With
this new struct we can easily update references for inline caches (but
not inline var caches as they just contain an int)
`(ulimit -n 30; make test-tool)` fails with unexplicit message:
"undefined method `write' for nil:NilClass" due to lack of stdout.
This change makes it explicit. [Bug #5577]
At the moment, there are some problems with regard to bundler + did_you_mean because of did_you_mean being a bundled gem. Since the vendored version of thor inside bundler and ruby itself explicitly requires did_you_mean, it can become difficult to load it when using Bundler.setup. See this issue: https://github.com/yuki24/did_you_mean/issues/117#issuecomment-482733159 for more details.
`exception_details` is not defined anywhere.
This commit fixes the following test crash in ruby 2.4
```
Error: test_assert_output_lines(TestFileUtils): NoMethodError: undefined method `exception_details' for #<TestFileUtils:0x00005556ef699178>
```
And replaces it with an actual test failure:
```
[Test::Unit::CoreAssertions::MiniTest::Assertion] exception expected, not #<NoMethodError: undefined method `full_message' for #<RuntimeError: ok>
Did you mean? message>.
```
https://github.com/ruby/fileutils/commit/2f38ba6e82
rename __builtin_inline!(code) to __builtin_cstmt(code).
Also this commit introduce the following inlining C code features.
* __builtin_cstmt!(STMT)
(renamed from __builtin_inline!)
Define a function which run STMT implicitly and call this function at
evatuation time. Note that you need to return some value in STMT.
If there is a local variables (includes method parameters), you can
read these values.
static VALUE func(ec, self) {
VALUE x = ...;
STMT
}
Usage:
def double a
# a is readable from C code.
__builtin_cstmt! 'return INT2FIX(FIX2INT(a) * 2);'
end
* __builtin_cexpr!(EXPR)
Define a function which invoke EXPR implicitly like `__builtin_cstmt!`.
Different from cstmt!, which compiled with `return EXPR;`.
(`return` and `;` are added implicitly)
static VALUE func(ec, self) {
VALUE x = ...;
return EXPPR;
}
Usage:
def double a
__builtin_cexpr! 'INT2FIX(FIX2INT(a) * 2)'
end
* __builtin_cconst!(EXPR)
Define a function which invoke EXPR implicitly like cexpr!.
However, the function is called once at compile time, not evaluated time.
Any local variables are not accessible (because there is no local variable
at compile time).
Usage:
GCC = __builtin_cconst! '__GNUC__'
* __builtin_cinit!(STMT)
STMT are writtein in auto-generated code.
This code does not return any value.
Usage:
__builtin_cinit! '#include <zlib.h>'
def no_compression?
__builtin_cconst! 'Z_NO_COMPRESSION ? Qtrue : Qfalse'
end
This removes the security features added by $SAFE = 1, and warns for access
or modification of $SAFE from Ruby-level, as well as warning when calling
all public C functions related to $SAFE.
This modifies some internal functions that took a safe level argument
to no longer take the argument.
rb_require_safe now warns, rb_require_string has been added as a
version that takes a VALUE and does not warn.
One public C function that still takes a safe level argument and that
this doesn't warn for is rb_eval_cmd. We may want to consider
adding an alternative method that does not take a safe level argument,
and warn for rb_eval_cmd.
Fixes [Bug #16332]
Constant access was changed to no longer allow top-level constant access
through `nil`, but `defined?` wasn't changed at the same time to stay
consistent.
Use a separate defined type to distinguish between a constant
referenced from the current lexical scope and one referenced from
another namespace.
by using ObjectSpace.trace_object_allocations.
`make test-all LEAK_CHECKER_TRACE_OBJECT_ALLOCATION=true` will print not
only leaked fds but also where it was created.
Add an experimental `__builtin_inline!(c_expression)` special intrinsic
which run a C code snippet.
In `c_expression`, you can access the following variables:
* ec (rb_execution_context_t *)
* self (const VALUE)
* local variables (const VALUE)
Not that you can read these variables, but you can not write them.
You need to return from this expression and return value will be a
result of __builtin_inline!().
Examples:
`def foo(x) __builtin_inline!('return rb_p(x);'); end` calls `p(x)`.
`def double(x) __builtin_inline!('return INT2NUM(NUM2INT(x) * 2);')`
returns x*2.
I noticed that some files in rubygems were executable, and I could think
of no reason why they should be.
In general, I think ruby files should never have the executable bit set
unless they include a shebang, so I run the following command over the
whole repo:
```bash
find . -name '*.rb' -type f -executable -exec bash -c 'grep -L "^#!" $1 || chmod -x $1' _ {} \;
```
http://ci.rvm.jp/results/trunk-mjit@silicon-docker/2380788
```
test_all #<Thread:0x000055b6c8e9fca8@/tmp/ruby/v2/src/trunk-mjit/tool/lib/test/unit/parallel.rb:42 run> terminated with exception (report_on_exception is true):
<internal:pack>:134:in `pack': no implicit conversion of false into String (TypeError)
from /tmp/ruby/v2/src/trunk-mjit/tool/lib/test/unit/parallel.rb:160:in `_report'
from /tmp/ruby/v2/src/trunk-mjit/tool/lib/test/unit/parallel.rb:45:in `block in _run_suite'
```
miniruby load *.rb from srcdir. To specify file path,
tool/mk_builtin_loader.rb embed full path of each *.rb file.
However it prevent to pre-generation of required files for tarball.
This patch generate srcdir/*.rb from __FILE__ information.
If there is a type mismatch between expected builtin function type
and actual function type, C compiler shows warning.
For example, `__builtin_func(1, 2)` expects
`func(rb_ec_t*, VALUE self, VALUE p1, VALUE p2)` function definition.
However, it is easy to overlook "warning" messages. So this patch
changes to stop compiling as an error if there is a mismatch.
Support loading builtin features written in Ruby, which implement
with C builtin functions.
[Feature #16254]
Several features:
(1) Load .rb file at boottime with native binary.
Now, prelude.rb is loaded at boottime. However, this file is contained
into the interpreter as a text format and we need to compile it.
This patch contains a feature to load from binary format.
(2) __builtin_func() in Ruby call func() written in C.
In Ruby file, we can write `__builtin_func()` like method call.
However this is not a method call, but special syntax to call
a function `func()` written in C. C functions should be defined
in a file (same compile unit) which load this .rb file.
Functions (`func` in above example) should be defined with
(a) 1st parameter: rb_execution_context_t *ec
(b) rest parameters (0 to 15).
(c) VALUE return type.
This is very similar requirements for functions used by
rb_define_method(), however `rb_execution_context_t *ec`
is new requirement.
(3) automatic C code generation from .rb files.
tool/mk_builtin_loader.rb creates a C code to load .rb files
needed by miniruby and ruby command. This script is run by
BASERUBY, so *.rb should be written in BASERUBY compatbile
syntax. This script load a .rb file and find all of __builtin_
prefix method calls, and generate a part of C code to export
functions.
tool/mk_builtin_binary.rb creates a C code which contains
binary compiled Ruby files needed by ruby command.
Prior to this changeset, majority of inline cache mishits resulted
into the same method entry when rb_callable_method_entry() resolves
a method search. Let's not call the function at the first place on
such situations.
In doing so we extend the struct rb_call_cache from 44 bytes (in
case of 64 bit machine) to 64 bytes, and fill the gap with
secondary class serial(s). Call cache's class serials now behavies
as a LRU cache.
Calculating -------------------------------------
ours 2.7 2.6
vm2_poly_same_method 2.339M 1.744M 1.369M i/s - 6.000M times in 2.565086s 3.441329s 4.381386s
Comparison:
vm2_poly_same_method
ours: 2339103.0 i/s
2.7: 1743512.3 i/s - 1.34x slower
2.6: 1369429.8 i/s - 1.71x slower
To perform a regular method call, the VM needs two structs,
`rb_call_info` and `rb_call_cache`. At the moment, we allocate these two
structures in separate buffers. In the worst case, the CPU needs to read
4 cache lines to complete a method call. Putting the two structures
together reduces the maximum number of cache line reads to 2.
Combining the structures also saves 8 bytes per call site as the current
layout uses separate two pointers for the call info and the call cache.
This saves about 2 MiB on Discourse.
This change improves the Optcarrot benchmark at least 3%. For more
details, see attached bugs.ruby-lang.org ticket.
Complications:
- A new instruction attribute `comptime_sp_inc` is introduced to
calculate SP increase at compile time without using call caches. At
compile time, a `TS_CALLDATA` operand points to a call info struct, but
at runtime, the same operand points to a call data struct. Instruction
that explicitly define `sp_inc` also need to define `comptime_sp_inc`.
- MJIT code for copying call cache becomes slightly more complicated.
- This changes the bytecode format, which might break existing tools.
[Misc #16258]
This reverts commit 43015275b9.
`assert_raise_message` in test-unit is different from
`assert_raise_with_message`. It checks the exception message
only, but not the exception class,
parallel test (`make test-all TESTS=-j8`) runs tests on specified
number of processes. However, some test checkers written in
`runner.rb` are not loaded. This fix enable these checkers on
parallel tests.
See also: https://github.com/ruby/ruby/pull/2508
Takashi Kokubun
Hiroshi SHIBATA
Nobuyoshi Nakada
David Rodríguez
卜部昌平
Koichi Sasada
Yusuke Endoh
Vít Ondruch
aycabta
NARUSE, Yui
Jun Aruga
Kazuhiro NISHIYAMA
Aaron Patterson
Alan Wu
Kevin Deisz
Jeremy Evans
Dylan Thacker-Smith
NAKAMURA Usaku