This commit splits gc.c into two files:
- gc.c now only contains code not specific to Ruby GC. This includes
code to mark objects (which the GC implementation may choose not to
use) and wrappers for internal APIs that the implementation may need
to use (e.g. locking the VM).
- gc_impl.c now contains the implementation of Ruby's GC. This includes
marking, sweeping, compaction, and statistics. Most importantly,
gc_impl.c only uses public APIs in Ruby and a limited set of functions
exposed in gc.c. This allows us to build gc_impl.c independently of
Ruby and plug Ruby's GC into itself.
The test assumes `:foo` is a static symbol, but that is only true
if a literal `:foo` was parsed before `"foo".to_sym` was evaled:
```ruby
require 'objspace'
foo_sym = "foo".to_sym
puts ObjectSpace.dump(eval(":foo"))
```
```
{"address":"0x100fb46d0", "type":"SYMBOL", "shape_id":10, "slot_size":40, "class":"0x100d3e9c8", "frozen":true, "bytesize":3, "value":"foo", "memsize":40, "flags":{"wb_protected":true, "marking":true, "marked":true}}
```
Since the callback defined in the objspace module might give up the GVL,
we need to make sure the right cr->mfd value is set back after the GVL
is re-obtained.
Some code out there blind calls `force_encoding` without checking
what the original encoding was, which clears the coderange uselessly.
If the String is big, it can be a rather costly mistake.
For instance the `rack-utf8_sanitizer` gem does this on request
bodies.
When a Ractor is created whilst a tracepoint for
RUBY_INTERNAL_EVENT_NEWOBJ is active, the interpreter crashes. This is
because during the early setup of the Ractor, the stdio objects are
created, which allocates Ruby objects, which fires the tracepoint.
However, the tracepoint machinery tries to dereference the control frame
(ec->cfp->pc), which isn't set up yet and so crashes with a null pointer
dereference.
Fix this by not firing GC tracepoints if cfp isn't yet set up.
We need to zero out the whole slot when running the newobj hook for a
newly allocated class because the slot could be filled with garbage,
which would cause a crash if a GC runs inside of the newobj hook.
For example, the following script crashes:
```
require "objspace"
GC.stress = true
ObjectSpace.trace_object_allocations {
100.times do
Class.new
end
}
```
[Bug #19482]
If the previous instruction is not a leaf instruction, then the PC was
incremented before the instruction was ran (meaning the currently
executing instruction is actually the previous instruction), so we
should not increment the PC otherwise we will calculate the source
line for the next instruction.
This bug can be reproduced in the following script:
```
require "objspace"
ObjectSpace.trace_object_allocations_start
a =
1.0 / 0.0
p [ObjectSpace.allocation_sourceline(a), ObjectSpace.allocation_sourcefile(a)]
```
Which outputs: [4, "test.rb"]
This is incorrect because the object was allocated on line 10 and not
line 4. The behaviour is correct when we use a leaf instruction (e.g.
if we replaced `1.0 / 0.0` with `"hello"`), then the output is:
[10, "test.rb"].
[Bug #19456]
```
{"address":"0x7f8c03e9fcf0", "type":"STRING", "shape_id":10, "slot_size":40, "class":"0x7f8c00dbed98", "frozen":true, "embedded":true, "fstring":true, "bytesize":5, "value":"TEST2", "encoding":"US-ASCII", "coderange":"7bit", "memsize":40, "flags":{"wb_protected":true}}
{"address":"0x7f8c03e9ffc0", "type":"STRING", "shape_id":0, "slot_size":40, "class":"0x7f8c00dbed98", "embedded":true, "bytesize":5, "value":"TEST2", "encoding":"US-ASCII", "coderange":"7bit", "memsize":40, "flags":{"wb_protected":true}}
{"address":"0x7f8c03e487c0", "type":"STRING", "shape_id":0, "slot_size":40, "class":"0x7f8c00dbed98", "embedded":true, "bytesize":5, "value":"TEST2", "encoding":"UTF-8", "coderange":"unknown", "file":"-", "line":4, "method":"dump_my_heap_please", "generation":1, "memsize":40, "flags":{"wb_protected":true}}
1) Failure:
TestObjSpace#test_dump_all [/tmp/ruby/src/trunk-gc-asserts/test/objspace/test_objspace.rb:622]:
number of strings.
<2> expected but was
<3>.
```
This failure only occurred on a ruby built with `DEFS=\"-DRGENGC_CHECK_MODE=2\""`
and only on a specific machine (Docker container) and difficult to reproduce,
so skip this failure to check other failures.
I see several arguments in doing so.
First they use a non trivial amount of memory, so for various memory
profiling/mapping tools it is relevant to have visibility of the space
occupied by shapes.
Then, some pathological code can create a tons of shape, so it is
valuable to have a way to have a way to observe shapes without having
to compile Ruby with `SHAPE_DEBUG=1`.
And additionally it's likely much faster to dump then this way than
to use `RubyVM::Shape`.
There are however a few open questions:
- Shapes can't respect the `since:` argument. Not sure what to do when
it is provided. Would probably make sense to not dump them.
- Maybe it would make more sense to have a separate `ObjectSpace.dump_shapes`?
- Maybe instead `dump_all` should take a `shapes: false` argument?
Additionally, `ObjectSpace.dump_shapes` is added for the use case of
debugging the evolution of the shape tree.
This commit adds RVALUE_OVERHEAD for storing metadata at the end of the
slot. This commit moves the ractor_belonging_id in debug builds from the
flags to RVALUE_OVERHEAD which frees the 16 bits in the headers for
object shapes.
This commit adds a `capacity` field to shapes, and adds shape
transitions whenever an object's capacity changes. Objects which are
allocated out of a bigger size pool will also make a transition from the
root shape to the shape with the correct capacity for their size pool
when they are allocated.
This commit will allow us to remove numiv from objects completely, and
will also mean we can guarantee that if two objects share shapes, their
IVs are in the same positions (an embedded and extended object cannot
share shapes). This will enable us to implement ivar sets in YJIT using
object shapes.
Co-Authored-By: Aaron Patterson <tenderlove@ruby-lang.org>
This patch pushes dummy frames when loading code for the
profiling purpose.
The following methods push a dummy frame:
* `Kernel#require`
* `Kernel#load`
* `RubyVM::InstructionSequence.compile_file`
* `RubyVM::InstructionSequence.load_from_binary`
https://bugs.ruby-lang.org/issues/18559
Object Shapes is used for accessing instance variables and representing the
"frozenness" of objects. Object instances have a "shape" and the shape
represents some attributes of the object (currently which instance variables are
set and the "frozenness"). Shapes form a tree data structure, and when a new
instance variable is set on an object, that object "transitions" to a new shape
in the shape tree. Each shape has an ID that is used for caching. The shape
structure is independent of class, so objects of different types can have the
same shape.
For example:
```ruby
class Foo
def initialize
# Starts with shape id 0
@a = 1 # transitions to shape id 1
@b = 1 # transitions to shape id 2
end
end
class Bar
def initialize
# Starts with shape id 0
@a = 1 # transitions to shape id 1
@b = 1 # transitions to shape id 2
end
end
foo = Foo.new # `foo` has shape id 2
bar = Bar.new # `bar` has shape id 2
```
Both `foo` and `bar` instances have the same shape because they both set
instance variables of the same name in the same order.
This technique can help to improve inline cache hits as well as generate more
efficient machine code in JIT compilers.
This commit also adds some methods for debugging shapes on objects. See
`RubyVM::Shape` for more details.
For more context on Object Shapes, see [Feature: #18776]
Co-Authored-By: Aaron Patterson <tenderlove@ruby-lang.org>
Co-Authored-By: Eileen M. Uchitelle <eileencodes@gmail.com>
Co-Authored-By: John Hawthorn <john@hawthorn.email>
Sidekiq has a method named `❨╯°□°❩╯︵┻━┻`which corrupts
heap dumps.
Normally we could just dump is as is since it's valid UTF-8 and need
no escaping. But our code to escape control characters isn't UTF-8
aware so it's more complicated than it seems.
Ultimately since the overwhelming majority of method names are
pure ASCII, it's not a big loss to just skip it.
I suspect that some shared pages are invalidated because
some static string don't have their coderange set eagerly.
So the first time they are scanned, the entire memory page is
invalidated.
Being able to see the coderange in `ObjectSpace` would help debug
this.
And in addition `dump` currently call `is_broken_string()` and `is_ascii_string()`
which both end up scanning the string and assigning coderange. I think it's
undesirable as `dump` should be read only.
Inside ObjectSpace.reachable_objects_from we keep an internal identhash
in order to de-duplicate reachable objects when wrapping them as
InternalObject. Previously this hash was not hidden, making it possible
to leak references to those internal objects to Ruby if using
ObjectSpace.each_object.
This commit solves this by hiding the hash. To simplify collection of
values, we instead now just use the hash as a set of visited objects,
and collect an Array (not hidden) of values to be returned.
Add a new macro BASE_SLOT_SIZE that determines the slot size.
For Variable Width Allocation (compiled with USE_RVARGC=1), all slot
sizes are powers-of-2 multiples of BASE_SLOT_SIZE.
For USE_RVARGC=0, BASE_SLOT_SIZE is set to sizeof(RVALUE).
When `require "objspace/trace"` fails, previously the failure says:
```
1) Failure:
TestObjSpace#test_objspace_trace [/tmp/ruby/v3/src/trunk-mjit/test/objspace/test_objspace.rb:621]:
<3> expected but was
<0>.
```
but this is hard to debug.
This file, when require'ed, starts tracing the object allocations, and
redefines `Kernel#p` to show the allocation site.
This commit is experimental; the library name and APIs may change.
[Feature #17762]
`memsize_of(Object.new)` can be changed with past ivar creation
history for Object instances (another Object instance has 4 or
more ivars, next created Object instance has the area for the
ivars). So use antoher class for the comparison.
test_memsize_of_iseq fails on repeat tests and it seems to difficult
to solve immediately. Now this test is skipped.
It seems that the result of `memsize_of(Object.new)` are increased.
Why...?
Peter Zhu
Jean Boussier
Nobuyoshi Nakada
Jean Boussier
Takashi Kokubun
KJ Tsanaktsidis
John Hawthorn
lukeg
Yusuke Endoh
Koichi Sasada
Jemma Issroff
Aaron Patterson
Jean byroot Boussier
Matt Valentine-House
git
Ryuta Kamizono