This patch contains several ideas:
(1) Disposable inline method cache (IMC) for race-free inline method cache
* Making call-cache (CC) as a RVALUE (GC target object) and allocate new
CC on cache miss.
* This technique allows race-free access from parallel processing
elements like RCU.
(2) Introduce per-Class method cache (pCMC)
* Instead of fixed-size global method cache (GMC), pCMC allows flexible
cache size.
* Caching CCs reduces CC allocation and allow sharing CC's fast-path
between same call-info (CI) call-sites.
(3) Invalidate an inline method cache by invalidating corresponding method
entries (MEs)
* Instead of using class serials, we set "invalidated" flag for method
entry itself to represent cache invalidation.
* Compare with using class serials, the impact of method modification
(add/overwrite/delete) is small.
* Updating class serials invalidate all method caches of the class and
sub-classes.
* Proposed approach only invalidate the method cache of only one ME.
See [Feature #16614] for more details.
Now, rb_call_info contains how to call the method with tuple of
(mid, orig_argc, flags, kwarg). Most of cases, kwarg == NULL and
mid+argc+flags only requires 64bits. So this patch packed
rb_call_info to VALUE (1 word) on such cases. If we can not
represent it in VALUE, then use imemo_callinfo which contains
conventional callinfo (rb_callinfo, renamed from rb_call_info).
iseq->body->ci_kw_size is removed because all of callinfo is VALUE
size (packed ci or a pointer to imemo_callinfo).
To access ci information, we need to use these functions:
vm_ci_mid(ci), _flag(ci), _argc(ci), _kwarg(ci).
struct rb_call_info_kw_arg is renamed to rb_callinfo_kwarg.
rb_funcallv_with_cc() and rb_method_basic_definition_p_with_cc()
is temporary removed because cd->ci should be marked.
On s390x, TestFiber#test_stack_size fails with SEGV.
https://rubyci.org/logs/rubyci.s3.amazonaws.com/rhel_zlinux/ruby-master/log/20200205T223421Z.fail.html.gz
```
TestFiber#test_stack_size [/home/chkbuild/build/20200205T223421Z/ruby/test/ruby/test_fiber.rb:356]:
pid 23844 killed by SIGABRT (signal 6) (core dumped)
| -e:1:in `times': stack level too deep (SystemStackError)
| from -e:1:in `rec'
| from -e:1:in `block (3 levels) in rec'
| from -e:1:in `times'
| from -e:1:in `block (2 levels) in rec'
| from -e:1:in `times'
| from -e:1:in `block in rec'
| from -e:1:in `times'
| from -e:1:in `rec'
| ... 172 levels...
| from -e:1:in `block in rec'
| from -e:1:in `times'
| from -e:1:in `rec'
| from -e:1:in `block in <main>'
| -e: [BUG] Segmentation fault at 0x0000000000000000
```
This change tries a similar fix with
ef64ab917e and
3ddbba84b5.
Saves comitters' daily life by avoid #include-ing everything from
internal.h to make each file do so instead. This would significantly
speed up incremental builds.
We take the following inclusion order in this changeset:
1. "ruby/config.h", where _GNU_SOURCE is defined (must be the very
first thing among everything).
2. RUBY_EXTCONF_H if any.
3. Standard C headers, sorted alphabetically.
4. Other system headers, maybe guarded by #ifdef
5. Everything else, sorted alphabetically.
Exceptions are those win32-related headers, which tend not be self-
containing (headers have inclusion order dependencies).
One day, I could not resist the way it was written. I finally started
to make the code clean. This changeset is the beginning of a series of
housekeeping commits. It is a simple refactoring; split internal.h into
files, so that we can divide and concur in the upcoming commits. No
lines of codes are either added or removed, except the obvious file
headers/footers. The generated binary is identical to the one before.
I am trying to fix this error:
http://ci.rvm.jp/results/trunk-gc_compact@silicon-docker/2491596
Somehow we have a page in the `free_pages` list that is full. This
commit refactors the code so that any time we add a page to the
`free_pages` list, we do it via `heap_add_freepage`. That function then
asserts that the free slots on that page are not 0.
This changeset makes no difference unless GC_DEBUG is on. When that flag is
set, struct RVALUE is bigger than struct RObject. We have to take care of the
additional fields. Otherwise we get a SIGSEGV like shown below.
The way obj is initialized in this patch works for both GC_DEBUG is on and off.
See also ISO/IEC 9899:1999 section 6.7.8 paragraph #21.
```
Program received signal SIGSEGV, Segmentation fault.
__strlen_avx2 () at ../sysdeps/x86_64/multiarch/strlen-avx2.S:62
62 ../sysdeps/x86_64/multiarch/strlen-avx2.S: No such file or directory
(gdb) bt
#0 __strlen_avx2 () at ../sysdeps/x86_64/multiarch/strlen-avx2.S:62
#1 0x00005555557dd9a7 in BSD_vfprintf (fp=0x7fffffff6be0, fmt0=0x5555558f3059 "@%s:%d", ap=0x7fffffff6dd0) at vsnprintf.c:1027
#2 0x00005555557db6f5 in ruby_do_vsnprintf (str=0x555555bfc58d <obj_info_buffers+1325> "", n=211, fmt=0x5555558f3059 "@%s:%d", ap=0x7fffffff6dd0) at sprintf.c:1022
#3 0x00005555557db909 in ruby_snprintf (str=0x555555bfc58d <obj_info_buffers+1325> "", n=211, fmt=0x5555558f3059 "@%s:%d") at sprintf.c:1040
#4 0x0000555555661ef4 in rb_raw_obj_info (buff=0x555555bfc560 <obj_info_buffers+1280> "0x0000555555d2bfa0 [0 ] T_STRING (String)", buff_size=256, obj=93825000456096) at gc.c:11449
#5 0x000055555565baaf in obj_info (obj=93825000456096) at gc.c:11612
#6 0x000055555565bae1 in rgengc_remembered (objspace=0x555555c0a1c0, obj=93825000456096) at gc.c:6618
#7 0x0000555555666987 in newobj_init (klass=93824999964192, flags=5, v1=0, v2=0, v3=0, wb_protected=1, objspace=0x555555c0a1c0, obj=93825000456096) at gc.c:2134
#8 0x0000555555666e49 in newobj_slowpath (klass=93824999964192, flags=5, v1=0, v2=0, v3=0, objspace=0x555555c0a1c0, wb_protected=1) at gc.c:2209
#9 0x0000555555666b94 in newobj_slowpath_wb_protected (klass=93824999964192, flags=5, v1=0, v2=0, v3=0, objspace=0x555555c0a1c0) at gc.c:2220
#10 0x000055555565751b in newobj_of (klass=93824999964192, flags=5, v1=0, v2=0, v3=0, wb_protected=1) at gc.c:2256
#11 0x00005555556575ca in rb_wb_protected_newobj_of (klass=93824999964192, flags=5) at gc.c:2272
#12 0x00005555557f36ea in str_alloc (klass=93824999964192) at string.c:728
#13 0x00005555557f2128 in rb_str_buf_new (capa=0) at string.c:1317
#14 0x000055555578c66d in rb_reg_preprocess (p=0x555555cc8148 "^-(.)(.+)?", end=0x555555cc8152 "", enc=0x555555cc7c80, fixed_enc=0x7fffffff74e8, err=0x7fffffff75f0 "") at re.c:2682
#15 0x000055555578ea13 in rb_reg_initialize (obj=93825000046736, s=0x555555cc8148 "^-(.)(.+)?", len=10, enc=0x555555cc7c80, options=0, err=0x7fffffff75f0 "", sourcefile=0x555555d1a5c0 "lib/optparse.rb", sourceline=1460) at re.c:2808
#16 0x000055555578e285 in rb_reg_initialize_str (obj=93825000046736, str=93825000046904, options=0, err=0x7fffffff75f0 "", sourcefile=0x555555d1a5c0 "lib/optparse.rb", sourceline=1460) at re.c:2869
#17 0x000055555578ee02 in rb_reg_compile (str=93825000046904, options=0, sourcefile=0x555555d1a5c0 "lib/optparse.rb", sourceline=1460) at re.c:2958
#18 0x0000555555748dfb in rb_parser_reg_compile (p=0x555555d1f760, str=93825000046904, options=0) at parse.y:12157
#19 0x00005555557581c3 in parser_reg_compile (p=0x555555d1f760, str=93825000046904, options=0) at parse.y:12151
#20 0x00005555557580ac in reg_compile (p=0x555555d1f760, str=93825000046904, options=0) at parse.y:12167
#21 0x0000555555746ebb in new_regexp (p=0x555555d1f760, node=0x555555dece68, options=0, loc=0x7fffffff89e8) at parse.y:10072
#22 0x000055555573d1f5 in ruby_yyparse (p=0x555555d1f760) at parse.y:4395
#23 0x000055555574a582 in yycompile0 (arg=93825000404832) at parse.y:5945
#24 0x00005555558c6898 in rb_suppress_tracing (func=0x55555574a470 <yycompile0>, arg=93825000404832) at vm_trace.c:427
#25 0x0000555555748290 in yycompile (vparser=93824999283456, p=0x555555d1f760, fname=93824999283624, line=1) at parse.y:5994
#26 0x00005555557481ae in rb_parser_compile_file_path (vparser=93824999283456, fname=93824999283624, file=93824999283400, start=1) at parse.y:6098
#27 0x00005555557cdd35 in load_file_internal (argp_v=140737488331760) at ruby.c:2023
#28 0x00005555556438c5 in rb_ensure (b_proc=0x5555557cd610 <load_file_internal>, data1=140737488331760, e_proc=0x5555557cddd0 <restore_load_file>, data2=140737488331760) at eval.c:1128
#29 0x00005555557cb68b in load_file (parser=93824999283456, fname=93824999283624, f=93824999283400, script=0, opt=0x7fffffffa468) at ruby.c:2142
#30 0x00005555557cb339 in rb_parser_load_file (parser=93824999283456, fname_v=93824999283624) at ruby.c:2164
#31 0x00005555556ba3e1 in load_iseq_eval (ec=0x555555c0a650, fname=93824999283624) at load.c:579
#32 0x00005555556b857a in require_internal (ec=0x555555c0a650, fname=93824999284352, exception=1) at load.c:1016
#33 0x00005555556b7967 in rb_require_string (fname=93824999284464) at load.c:1105
#34 0x00005555556b7939 in rb_f_require (obj=93824999994824, fname=93824999284464) at load.c:811
#35 0x00005555558b7ae0 in call_cfunc_1 (recv=93824999994824, argc=1, argv=0x7ffff7ecd0a8, func=0x5555556b7920 <rb_f_require>) at vm_insnhelper.c:2348
#36 0x00005555558a8889 in vm_call_cfunc_with_frame (ec=0x555555c0a650, reg_cfp=0x7ffff7fccfa0, calling=0x7fffffffaab0, cd=0x555555d76a10, empty_kw_splat=0) at vm_insnhelper.c:2513
#37 0x000055555589fb5c in vm_call_cfunc (ec=0x555555c0a650, reg_cfp=0x7ffff7fccfa0, calling=0x7fffffffaab0, cd=0x555555d76a10) at vm_insnhelper.c:2538
#38 0x000055555589f22e in vm_call_method_each_type (ec=0x555555c0a650, cfp=0x7ffff7fccfa0, calling=0x7fffffffaab0, cd=0x555555d76a10) at vm_insnhelper.c:2924
#39 0x000055555589ef47 in vm_call_method (ec=0x555555c0a650, cfp=0x7ffff7fccfa0, calling=0x7fffffffaab0, cd=0x555555d76a10) at vm_insnhelper.c:3038
#40 0x0000555555866dbd in vm_call_general (ec=0x555555c0a650, reg_cfp=0x7ffff7fccfa0, calling=0x7fffffffaab0, cd=0x555555d76a10) at vm_insnhelper.c:3075
#41 0x00005555558ae557 in vm_sendish (ec=0x555555c0a650, reg_cfp=0x7ffff7fccfa0, cd=0x555555d76a10, block_handler=0, method_explorer=0x5555558ae5d0 <vm_search_method_wrap>) at vm_insnhelper.c:4021
#42 0x000055555587745b in vm_exec_core (ec=0x555555c0a650, initial=0) at insns.def:801
#43 0x0000555555899b9c in rb_vm_exec (ec=0x555555c0a650, mjit_enable_p=1) at vm.c:1907
#44 0x000055555589aaf0 in rb_iseq_eval_main (iseq=0x555555c1da80) at vm.c:2166
#45 0x0000555555641f0b in rb_ec_exec_node (ec=0x555555c0a650, n=0x555555c1da80) at eval.c:277
#46 0x0000555555641d62 in ruby_run_node (n=0x555555c1da80) at eval.c:335
#47 0x000055555557a188 in main (argc=11, argv=0x7fffffffc848) at main.c:50
(gdb) fr 7
#7 0x0000555555666987 in newobj_init (klass=93824999964192, flags=5, v1=0, v2=0, v3=0, wb_protected=1, objspace=0x555555c0a1c0, obj=93825000456096) at gc.c:2134
2134 if (rgengc_remembered(objspace, (VALUE)obj)) rb_bug("newobj: %s is remembered.", obj_info(obj));
(gdb) p ((struct RVALUE*)obj)->file
$1 = 0x65a5992b0fb25ce7 <error: Cannot access memory at address 0x65a5992b0fb25ce7>
(gdb)
```
Introduce new RUBY_DEBUG option 'ci' to inform Ruby interpreter
that an interpreter is running on CI environment.
With this option, `rb_bug()` shows more information includes
method entry information, local variables information for each
control frame.
Decades ago, among all the data that a class has, its method
table was no doubt the most frequently accessed data. Previous
data structures were based on that assumption.
Today that is no longer true. The most frequently accessed field
moved to class_serial. That field is not always as wide as VALUE
but if it is, let us swap m_tbl and class_serial.
Calculating -------------------------------------
ours trunk
Optcarrot Lan_Master.nes 47.363 46.630 fps
Comparison:
Optcarrot Lan_Master.nes
ours: 47.4 fps
trunk: 46.6 fps - 1.02x slower
This is significantly faster than checking BUILTIN_TYPEs because we
access significantly less memory. We also use popcount to count entire
words at a time.
The only functional difference from the previous implementation is that
T_ZOMBIE objects will no longer be counted. However those are temporary
objects which should be small in number, and this method has always been
an estimate.
Previously we would count the pinned objects on each comparison. Since
sorting is O(N log N) and we calculated this on both left and right
pages on each comparison this resulted in a extra iterations over the
slots.
These functions are used from within a compilation unit so we can
make them static, for better binary size. This changeset reduces
the size of generated ruby binary from 26,590,128 bytes to
26,584,472 bytes on my macihne.
This removes the related tests, and puts the related specs behind
version guards. This affects all code in lib, including some
libraries that may want to support older versions of Ruby.
This removes the security features added by $SAFE = 1, and warns for access
or modification of $SAFE from Ruby-level, as well as warning when calling
all public C functions related to $SAFE.
This modifies some internal functions that took a safe level argument
to no longer take the argument.
rb_require_safe now warns, rb_require_string has been added as a
version that takes a VALUE and does not warn.
One public C function that still takes a safe level argument and that
this doesn't warn for is rb_eval_cmd. We may want to consider
adding an alternative method that does not take a safe level argument,
and warn for rb_eval_cmd.
Previously we were passing the memory_id. This was broken previously if
compaction was run (which changes the memory_id) and now that object_id
is a monotonically increasing number it was always broken.
This commit fixes this by defering removal from the object_id table
until finalizers have run (for objects with finalizers) and also copying
the SEEN_OBJ_ID flag onto the zombie objects.
This changes object_id from being based on the objects location in
memory (or a nearby memory location in the case of a conflict) to be
based on an always increasing number.
This number is a Ruby Integer which allows it to overflow the size of a
pointer without issue (very unlikely to happen in real programs
especially on 64-bit, but a nice guarantee).
This changes obj_to_id_tbl and id_to_obj_tbl to both be maps of Ruby
objects to Ruby objects (previously they were Ruby object to C integer)
which simplifies updating them after compaction as we can run them
through gc_update_table_refs.
Co-authored-by: Aaron Patterson <tenderlove@ruby-lang.org>
This changes object_id from being based on the objects location in
memory (or a nearby memory location in the case of a conflict) to be
based on an always increasing number.
This number is a Ruby Integer which allows it to overflow the size of a
pointer without issue (very unlikely to happen in real programs
especially on 64-bit, but a nice guarantee).
This changes obj_to_id_tbl and id_to_obj_tbl to both be maps of Ruby
objects to Ruby objects (previously they were Ruby object to C integer)
which simplifies updating them after compaction as we can run them
through gc_update_table_refs.
Co-authored-by: Aaron Patterson <tenderlove@ruby-lang.org>
This commit is to attempt fixing this error:
http://ci.rvm.jp/results/trunk-gc-asserts@ruby-sky1/2353281
Each non-full heap_page struct contains a reference to the next page
that contains free slots. Compaction could fill any page, including
pages that happen to be linked to as "pages which contain free slots".
To fix this, we'll iterate each page, and rebuild the "free page list"
depending on the number of actual free slots on that page. If there are
no free slots on the page, we'll set the free_next pointer to NULL.
Finally we'll pop one page off the "free page list" and set it as the
"using page" for the next allocation.
This commit is to attempt fixing this error:
http://ci.rvm.jp/results/trunk-gc-asserts@ruby-sky1/2353281
Each non-full heap_page struct contains a reference to the next page
that contains free slots. Compaction could fill any page, including
pages that happen to be linked to as "pages which contain free slots".
To fix this, we'll iterate each page, and rebuild the "free page list"
depending on the number of actual free slots on that page. If there are
no free slots on the page, we'll set the free_next pointer to NULL.
Finally we'll pop one page off the "free page list" and set it as the
"using page" for the next allocation.
When we compact the heap, various st tables are updated, particularly
the table that contains the object id map. Updating an st table can
cause a GC to occur, and we need to prevent any GC from happening while
moving or updating references.
This reverts commit 60a7f9f446.
We can't have Ruby objects pointing at T_ZOMBIE objects otherwise we get
an error in the GC. We need to find a different way to update
references.
When we run finalizers we have to copy all of the finalizers to a new
data structure because a finalizer could add another finalizer and we
need to keep draining the "real" finalizer table until it's empty.
We don't want Ruby programs to mutate the finalizers that we're
iterating over as well.
Before this commit we would copy the finalizers in to a linked list.
The problem with this approach is that if compaction happens, the linked
list will need to be updated. But the GC doesn't know about the
existence of the linked list, so it could not update references. This
commit changes the linked list to be a Ruby array so that when
compaction happens, the arrays will automatically be updated and all
references remain valid.
Simple comparison between proc/ifunc/method invocations:
```
proc 15.209M (± 1.6%) i/s - 76.138M in 5.007413s
ifunc 15.195M (± 1.7%) i/s - 76.257M in 5.020106s
method 9.836M (± 1.2%) i/s - 49.272M in 5.009984s
```
As `proc` and `ifunc` have no significant difference, chosen the
latter for arity check.
Requested by ko1 that ability of calling rb_raise from anywhere
outside of GVL is "too much". Give up that part, move the GVL
aquisition routine into gc.c, and make our new gc_raise().
Koichi Sasada
卜部昌平
Yusuke Endoh
Nobuyoshi Nakada
Lourens Naudé
Kenta Murata
Aaron Patterson
John Hawthorn
Jeremy Evans