The name is misleading, as it seems like the function checks whether the
object is swept or not. But the function only checks whether the page is
before or after sweeping.
The FL_FINALIZE flag is set when there are finalizers for the object. We
can improver performance by not looking up in the table if the flag is
not set.
Using the following C extension:
#include "ruby/ruby.h"
static void data_free(void *_ptr) {}
static const rb_data_type_t data_type = {
"my_type",
{
NULL,
data_free,
},
0, 0, 0
};
static VALUE data_alloc(VALUE klass) {
return TypedData_Wrap_Struct(klass, &data_type, (void *)1);
}
void Init_myext(void) {
VALUE my_klass = rb_define_class("MyClass", rb_cObject);
rb_define_alloc_func(my_klass, data_alloc);
}
And the following benchmark:
require "benchmark"
final_objs = 1_000_000.times.map do
o = Object.new
ObjectSpace.define_finalizer(o, proc {})
o
end
puts(Benchmark.measure do
100_000_000.times do
MyClass.new
end
end)
Before:
10.974190 0.355037 11.329227 ( 11.416772)
After:
7.664310 0.347598 8.011908 ( 8.268969)
Rather than exposing that an imemo has a flag and four fields, this
changes the implementation to only expose one field (the klass) and
fills the rest with 0. The type will have to fill in the values themselves.
The switch statement is not exhaustive, meaning the "unreachable"
comment was not correct. This commit fixes it by making the list
exhaustive and adding an rb_bug in the default case.
Previously every call to vm_ci_new (when the CI was not packable) would
result in a different callinfo being returned this meant that every
kwarg callsite had its own CI.
When calling, different CIs result in different CCs. These CIs and CCs
both end up persisted on the T_CLASS inside cc_tbl. So in an eval loop
this resulted in a memory leak of both types of object. This also likely
resulted in extra memory used, and extra time searching, in non-eval
cases.
For simplicity in this commit I always allocate a CI object inside
rb_vm_ci_lookup, but ideally we would lazily allocate it only when
needed. I hope to do that as a follow up in the future.
rb_objspace_call_finalizer didn't free fibers and neither did
rb_objspace_free_objects, which caused fibers to be reported as leaked
when using RUBY_FREE_AT_EXIT. This commit changes rb_objspace_free_objects
to free all remaining Ruby objects.
This returns whether or not _any_ piece of memory in the range is
poisoned, not if _all_ of it is. That means that currently, with ASAN
enabled, pages which contain a single poisoned object are skipped
entirely from being iterated with objspace_each* family of functions.
[Bug #20220]
ASAN leaves a pointer to the fake frame on the stack; we can use the
__asan_addr_is_in_fake_stack API to work out the extent of the fake
stack and thus mark any VALUEs contained therein.
[Bug #20001]
T_DATA without a pointer or free function may still have ivars set on
them that need to be freed. The following leaked generic ivars for
example:
converter = Encoding::Converter.allocate
converter.instance_variable_set(:@foo, 1)
STACK OF 1 INSTANCE OF 'ROOT LEAK: <malloc in objspace_xmalloc0>':
<snip>
12 miniruby 0x10286ec50 ivar_set + 140 variable.c:1850
11 miniruby 0x102876afc generic_ivar_set + 136 variable.c:1668
ASAN leaves a pointer to the fake frame on the stack; we can use the
__asan_addr_is_in_fake_stack API to work out the extent of the fake
stack and thus mark any VALUEs contained therein.
[Bug #20001]
I was trying to debug an (unrelated) issue in the GC, and wanted to turn
on the trace-level GC output by compiling it with -DRGENGC_DEBUG=5.
Unfortunately, this actually causes a crash in newobj_init() because the
code there tries to log the obj_info() of the newly created object.
However, the object is not actually sufficiently set up for some of the
things that obj_info() tries to do:
* The instance variable table for a class is not yet initialized, and
when using variable-length RVALUES, said ivar table is embedded in
as-yet unitialized memory after the struct RValue. Attempting to read
this, as obj_info() does, causes a crash.
* T_DATA variables need to dereference their ->type field to print out
the underlying C type name, which is not set up until newobj_fill() is
called.
To fix this, create a new method `obj_info_basic`, which dumps out only
the parts of the object that are valid before the object is fully
initialized.
[Fixes#18795]
The for loops for marking and reference updating declaratively marked
TypedData objects did not mark/reference update the very last element.
When RGENGC_CHECK_MODE is turned on, this caused the test in Enumerator
to fail with:
tool/lib/test/unit/testcase.rb:173:in `rescue in run': failed to allocate memory (NoMemoryError)
This commit adds `GC.auto_compact = :empty` which will run
auto-compaction sorting pages by empty slots so the most amount of
objects will be moved. This will make it easier to write tests for
auto-compaction.
pinned_slots is not being reset every GC, which causes this assertion to
fail:
```
Assertion Failed: gc.c:7076:gc_pin:GET_HEAP_PAGE(obj)->pinned_slots <= GET_HEAP_PAGE(obj)->total_slots
```
This commit changes it to reset it at the beginning of every compaction
GC cycle.
Previously with RUBY_FREE_ON_EXIT, ractors where being xfree-ed which is incorrect since they are not xmalloced.
Instead we can free ractors with ractor free during shutdown. This change only effects main ractor freeing when RUBY_FREE_ON_EXIT is set.
Co-authored-by: John Hawthorn <john@hawthorn.email>
Previously, T_DATA and T_FILE objects did not have their instance
variables freed on exit which would be reported as a memory leak with
RUBY_FREE_ON_EXIT. This commit changes it to use obj_free which also
frees the generic instance variables.
Co-authored-by: Alan Wu <XrXr@users.noreply.github.com>
When compiling with cppflags=-DRGENGC_CHECK_MODE, the following crashes:
```
$ RUBY_FREE_ON_EXIT=1 ./miniruby -e 0
-e: [BUG] obj_free: RVALUE_MARKED(0x0000000103570020 [3LM ] T_CLASS (anon)) != FALSE
```
This commit clears the mark bits when rb_free_on_exit is enabled.
Our current implementation of rb_postponed_job_register suffers from
some safety issues that can lead to interpreter crashes (see bug #1991).
Essentially, the issue is that jobs can be called with the wrong
arguments.
We made two attempts to fix this whilst keeping the promised semantics,
but:
* The first one involved masking/unmasking when flushing jobs, which
was believed to be too expensive
* The second one involved a lock-free, multi-producer, single-consumer
ringbuffer, which was too complex
The critical insight behind this third solution is that essentially the
only user of these APIs are a) internal, or b) profiling gems.
For a), none of the usages actually require variable data; they will
work just fine with the preregistration interface.
For b), generally profiling gems only call a single callback with a
single piece of data (which is actually usually just zero) for the life
of the program. The ringbuffer is complex because it needs to support
multi-word inserts of job & data (which can't be atomic); but nobody
actually even needs that functionality, really.
So, this comit:
* Introduces a pre-registration API for jobs, with a GVL-requiring
rb_postponed_job_prereigster, which returns a handle which can be
used with an async-signal-safe rb_postponed_job_trigger.
* Deprecates rb_postponed_job_register (and re-implements it on top of
the preregister function for compatability)
* Moves all the internal usages of postponed job register
pre-registration
when the RUBY_FREE_ON_SHUTDOWN environment variable is set, manually free memory at shutdown.
Co-authored-by: Nobuyoshi Nakada <nobu@ruby-lang.org>
Co-authored-by: Peter Zhu <peter@peterzhu.ca>
need_major_gc is set when a major GC is required. However, if
gc_stress_no_major is also set, then it will not actually run a major
GC.
For example, the following script will sometimes crash:
```
GC.stress = 1
50000.times.map { [] }
```
With the following message:
```
[BUG] cannot create a new page after major GC
```
The intention of GC.verify_compaction_references is, I believe, to force
every single movable object to be moved, so that it's possible to debug
native extensions which not correctly updating their references to
objects they mark as movable.
To do this, it doubles the number of allocated pages for each size pool,
and sorts the heap pages so that the free ones are swept first; thus,
every object in an old page should be moved into a free slot in one of
the new pages.
This worked fine until movement of objects _between_ size pools during
compaction was implemented. That causes some problems for
verify_compaction_references:
* We were doubling the number of pages in each size pool, but actually
if some objects need to move into a _different_ pool, there's no
guarantee that they'll be enough room in that one.
* It's possible for the sweep & compact cursors to meet in one size pool
before all the objects that want to move into that size pool from
another are processed by the compaction.
You can see these problems by changing some of the movement tests in
test_gc_compact.rb to try and move e.g. 50,000 objects instead of
500; the test is not able to actually move all of the objects in a
single compaction run.
To fix this, we do two things in verify_compaction_references:
* Firstly, we add enough pages to every size pool to make them the same
size. This ensures that their compact cursors will all have space to
move during compaction (even if that means empty pages are
pointlessly compacted)
* Then, we examine every object and determine where it _wants_ to be
compacted into. We use this information to add additional pages to
each size pool to handle all objects which should live there.
With these two changes, we can move arbitrary amounts of objects into
the correct size pool in a single call to verify_compaction_references.
My _motivation_ for performing this work was to try and fix some test
stability issues in test_gc_compact.rb. I now no longer think that we
actually see this particular bug in rubyci.org, but I also think
verify_compaction_references should do what it says on the tin, so it's
worth keeping.
[Bug #20022]
This works like objspace_each_obj, except instead of being called with
the start & end address of each page, it's called with the page
structure itself.
[Bug #20022]
Objects with the same shape must always have the same "embeddedness"
(either embedded or heap allocated) because YJIT assumes so. However,
using remove_instance_variable, it's possible that some objects are
embedded and some are heap allocated because it does not re-embed heap
allocated objects.
This commit changes remove_instance_variable to re-embed Object
instance variables when it becomes small enough.
Embedded shared strings cannot be moved because strings point into the
slot of the shared string. There may be code using the RSTRING_PTR on
the stack, which would pin the string but not pin the shared string,
causing it to move.
When generic instance variable has a shape, it is marked movable. If it
it transitions to too complex, it needs to update references otherwise
it may have incorrect references.
This is required for the same reason that super CC needs it.
See 36023d5cb7.
Reproducer:
def cached_foo_callsite(obj) = obj.foo
class Foo
def foo = :v1
module R
refine Foo do
def foo = :unused
end
end
end
obj = Foo.new
cached_foo_callsite(obj) # set up cc with cme for foo=:v1
class Foo
def foo = :v2
end
GC.start # cme for foo=:v1 collected, if not reachable by cached_foo_callsite
cached_foo_callsite(obj)
[Bug #19994]
On large Ruby applications, shutdown may be slow if a major GC has just
started because rb_objspace_call_finalizer completes the GC.
This commit adds gc_abort which discards the mark stack if during
incremental marking and stops sweeping if during lazy sweeping.
Previously, because gc_update_object_references() did not update the
VALUEs in the too_complex ivar st_table for T_CLASS and T_MODULE
objects, GC compaction could finish with corrupted objects.
- start with `klass`, not too_complex
- GC incremental step marks `klass` and its ivars
- ruby code makes `klass` too_complex
- GC compaction runs and move `klass` ivars, but because `klass` is
too_complex, its ivars are not updated by gc_update_object_references(),
leaving T_NONE or T_MOVED objects in the ivar table.
Co-authored-by: Peter Zhu <peter@peterzhu.ca>
Marking both keys and values versus marking just values is an important
distinction, but previously, gc_update_tbl_refs() and gc_update_table_refs()
had names that were too similar.
The st_table storing ivars for too_complex T_OBJECTs have IDs as keys,
but we were marking the IDs unnecessary previously, maybe due to the
confusing naming.
Previously, it tripped the assert about too_complex in
ROBJECT_IV_CAPACITY(). This fixes double faults for some crashes and
helps with use during development.
Since the callback defined in the objspace module might give up the GVL,
we need to make sure the right cr->mfd value is set back after the GVL
is re-obtained.
The previous implementation was using the pointer given
by `DATA_PTR` in all cases. But in the case of an embedded
TypedData, that pointer is garbage, we need to use RTYPEDDATA_GET_DATA
to get the proper data pointer.
Co-Authored-By: Étienne Barrié <etienne.barrie@gmail.com>
This commit adds a new flag RUBY_TYPED_EMBEDDABLE that allows the data
of a TypedData object to be embedded after the object itself. This will
improve cache locality and allow us to save the 8 byte data pointer.
Co-Authored-By: Jean Boussier <byroot@ruby-lang.org>
Tracks other callinfo that references the same kwargs and frees them when all references are cleared.
[bug #19906]
Co-authored-by: Peter Zhu <peter@peterzhu.ca>
fix memory leak in vm_method
This introduces a unified reference_count to clarify who is referencing a method.
This also allows us to treat the refinement method as the def owner since it counts itself as a reference
Co-authored-by: Peter Zhu <peter@peterzhu.ca>
By compacting into slots with pinned objects first, we improve the
efficiency of compaction. As it is less likely that there will exist
pages containing only pinned objects after compaction. This will
increase the number of free pages left after compaction and enable us to
free them.
This used to be the default compaction method before it was removed
(inadvertently?) during the introduction of auto_compaction.
This commit will sort the pages by the pinned slot count at the start of
a major GC that has been triggered by explicitly calling GC.compact (and
thus setting objspace->flags.during_compaction).
It works using the same method by which we sort the heap by empty slot
count during GC.verify_compaction_references.
Previously it was only being sorted during the verify compaction
references stage - so would only happen during testing.
This commit allows us to sort the heap prior to each explicit GC.compact
run
Previously, configuring any GC event hook would cause all allocations to
go through the newobj slowpath. We should only need to do that when the
newobj specifically is subscribed to.
This renames flags.has_hook to flags.has_newobj_hook, to make this new
usage clear. newobj_of0 was the only place which previously checked this
flag.
This should help fix the following flaky test:
```
1) Failure:
TestProcess#test_warmup_frees_pages [test/ruby/test_process.rb:2751]:
<0> expected but was
<1>.
```
If we're during incremental marking, then Ruby code can execute that
deallocates certain memory buffers that have been called with
rb_gc_mark_weak, which can cause use-after-free bugs.
The term "shady object" was renamed to "uncollectible write barrier
unprotected object", so rename `has_uncollectible_shady_objects` to
`has_uncollectible_wb_unprotected_objects` for consistency.
We always sweep at least 2048 slots per sweep step, but only pool one
page. For large size pools, 2048 slots is many pages but one page is
very few slots. This commit changes it so that at least 1024 slots are
placed in the pooled pages per sweep step.
We move all pooled pages to free pages at the start of incremental
marking, so we shouldn't run incremental marking only when we have run
out of free pages. This causes incremental marking to always complete
in a single step.
If we are in a minor GC and the object to mark is old, then the old
object should already be marked and cannot be reclaimed in this GC cycle
so we don't need to add it to the weak refences list.
This is an internal only function not exposed to the C extension API.
It's only use so far is from rb_vm_mark, where it's used to mark the
values in the vm->trap_list.cmd array.
There shouldn't be any reason why these cannot move.
This commit allows them to move by updating their references during the
reference updating step of compaction.
To do this we've introduced another internal function
rb_gc_update_values as a partner to rb_gc_mark_values.
This allows us to refactor rb_gc_mark_values to not pin
The old algorithm could calculate an undercount for the initial pages
due to two issues:
1. It did not take into account that some heap pages will have one less
slot due to alignment. It assumed that every heap page would be able
to be fully filled with slots. Pages that are unaligned with the slot
size will lose one slot. The new algorithm assumes that every page
will be unaligned.
2. It performed integer division, which truncates down. This means that
the number of pages might not actually satisfy the number of slots.
This can cause the heap to grow in `gc_sweep_finish_size_pool` after
allocating all of the allocatable pages because the total number of
slots would be less than the initial configured number of slots.
This commit changes RUBY_GC_HEAP_INIT_SIZE_{40,80,160,320,640}_SLOTS to
RUBY_GC_HEAP_{0,1,2,3,4}_INIT_SLOTS. This is easier to use because the
user does not need to determine the slot sizes (which can vary between
32 and 64 bit systems). They now just use the heap names
(`GC.stat_heap.keys`).
If initial slots is set, then during a minor GC, if we have allocatable
pages but the heap is mostly full, then we will set `grow_heap` to true
since `total_slots` does not count allocatable pages so it will be less
than `init_slots`. This can cause `allocatable_pages` to grow to much
higher than desired since it will appear that the heap is mostly full.
This commit adds `free_empty_pages` which frees all empty heap pages and
moves the number of pages freed to the allocatable pages counter. This
is used in Process.warmup to improve performance because page
invalidation from copy-on-write is slower than allocating a new page.
[Feature #19783]
This commit adds stats about weak references to `GC.latest_gc_info`.
It adds the following two keys:
- `weak_references_count`: number of weak references registered during
the last GC.
- `retained_weak_references_count`: number of weak references that
survived the last GC.
[Feature #19783]
This commit adds support for weak references in the GC through the
function `rb_gc_mark_weak`. Unlike strong references, weak references
does not mark the object, but rather lets the GC know that an object
refers to another one. If the child object is freed, the pointer from
the parent object is overwritten with `Qundef`.
Co-Authored-By: Jean Boussier <byroot@ruby-lang.org>
This commit adds key force_incremental_marking_finish_count to
GC.stat_heap. This statistic returns the number of times the size pool
has forced incremental marking to finish due to running out of slots.
Functions rgengc_remembered, rgengc_remembered_sweep, and
rgengc_remembersetbits_get are just wrappers of RVALUE_REMEMBERED and
doesn't do much more. We can remove all those and use RVALUE_REMEMBERED
directly instead.
We don't need to check stack for moved objects after compaction because
the mutator cannot run between marking the stack and the end of
compaction. However, the stack may have moved objects leftover from
marking and sweeping phases. This means that their pages will be
invalidated and all objects moved back. We don't need to move these
objects back.
This also fixes the issue on Windows where some compaction tests
sometimes fail due to the page of the object being invalidated.
This commit stores the initial slots per size pool, configured with
the environment variables `RUBY_GC_HEAP_INIT_SIZE_%d_SLOTS`. This
ensures that the configured initial slots remains a low bound for the
number of slots in the heap, which can prevent heaps from thrashing in
size.
From Ruby 3.0, refined method invocations are slow because
resolved methods are not cached by inline cache because of
conservertive strategy. However, `using` clears all caches
so that it seems safe to cache resolved method entries.
This patch caches resolved method entries in inline cache
and clear all of inline method caches when `using` is called.
fix [Bug #18572]
```ruby
# without refinements
class C
def foo = :C
end
N = 1_000_000
obj = C.new
require 'benchmark'
Benchmark.bm{|x|
x.report{N.times{
obj.foo; obj.foo; obj.foo; obj.foo; obj.foo;
obj.foo; obj.foo; obj.foo; obj.foo; obj.foo;
obj.foo; obj.foo; obj.foo; obj.foo; obj.foo;
obj.foo; obj.foo; obj.foo; obj.foo; obj.foo;
}}
}
_END__
user system total real
master 0.362859 0.002544 0.365403 ( 0.365424)
modified 0.357251 0.000000 0.357251 ( 0.357258)
```
```ruby
# with refinment but without using
class C
def foo = :C
end
module R
refine C do
def foo = :R
end
end
N = 1_000_000
obj = C.new
require 'benchmark'
Benchmark.bm{|x|
x.report{N.times{
obj.foo; obj.foo; obj.foo; obj.foo; obj.foo;
obj.foo; obj.foo; obj.foo; obj.foo; obj.foo;
obj.foo; obj.foo; obj.foo; obj.foo; obj.foo;
obj.foo; obj.foo; obj.foo; obj.foo; obj.foo;
}}
}
__END__
user system total real
master 0.957182 0.000000 0.957182 ( 0.957212)
modified 0.359228 0.000000 0.359228 ( 0.359238)
```
```ruby
# with using
class C
def foo = :C
end
module R
refine C do
def foo = :R
end
end
N = 1_000_000
using R
obj = C.new
require 'benchmark'
Benchmark.bm{|x|
x.report{N.times{
obj.foo; obj.foo; obj.foo; obj.foo; obj.foo;
obj.foo; obj.foo; obj.foo; obj.foo; obj.foo;
obj.foo; obj.foo; obj.foo; obj.foo; obj.foo;
obj.foo; obj.foo; obj.foo; obj.foo; obj.foo;
}}
}
`cc->klass` and `cc->cme_` can be free'ed while last marking
so that it should be checked bofore updating the pointers.
Note that `T_MOVED` is living, but `is_live_object()` returns false.
cc is callcache.
cc->klass (klass) should not be marked because if the klass is
free'ed, the cc->klass will be cleared by `vm_cc_invalidate()`.
cc->cme (cme) should not be marked because if cc is invalidated
when cme is free'ed.
- klass marks cme if klass uses cme.
- caller classe's ccs->cme marks cc->cme.
- if cc is invalidated (klass doesn't refer the cc),
cc is invalidated by `vm_cc_invalidate()` and cc->cme is
not be accessed.
- On the multi-Ractors, cme will be collected with global GC
so that it is safe if GC is not interleaving while accessing
cc and cme.
fix [Bug #19436]
```ruby
10_000.times{|i|
# p i if (i%1_000) == 0
str = "x" * 1_000_000
def str.foo = nil
eval "def call#{i}(s) = s.foo"
send "call#{i}", str
}
```
Without this patch:
```
real 1m5.639s
user 0m6.637s
sys 0m58.292s
```
and with this patch:
```
real 0m2.045s
user 0m1.627s
sys 0m0.164s
```
This both save time for when it will be eventually needed,
and avoid mutating heap pages after a potential fork.
Instrumenting some large Rails app, I've witnessed up to
58% of String instances having their coderange still unknown.
Peter Zhu
John Hawthorn
Alan Wu
KJ Tsanaktsidis
KJ Tsanaktsidis
Nobuyoshi Nakada
Rian McGuire
HParker
Koichi Sasada
Adam Hess
Soutaro Matsumoto
Aaron Patterson
Jean Boussier
Jean Boussier
Yusuke Endoh
Matt Valentine-House
eileencodes
Takashi Kokubun
Ruby
Kunshan Wang