I tried to build Ruby on a system without libyaml today and realized
that my attempt from <https://github.com/ruby/psych/pull/557> doesn't
fix the error in <https://github.com/ruby/psych/issues/552>. I still got
the same `LoadError` from `digest` which stopped the build.
Since `LoadError` is not a `StandardError`, a plain `rescue` doesn't catch
it. Catch `LoadError` explicitly instead and reduce the scope of the
`begin` block.
I tested this change in a Ruby build on macOS without libyaml installed
and confirmed that `make` continues with a warning instead of aborting:
*** Following extensions are not compiled:
psych:
Could not be configured. It will not be installed.
...
This should address <https://bugs.ruby-lang.org/issues/18790>.
https://github.com/ruby/psych/commit/251289ba83
This commit makes YJIT allocate memory for generated code gradually as
needed. Previously, YJIT allocates all the memory it needs on boot in
one go, leading to higher than necessary resident set size (RSS) and
time spent on boot initializing the memory with a large memset().
Users should no longer need to search for a magic number to pass to
`--yjit-exec-mem` since physical memory consumption should now more
accurately reflect the requirement of the workload.
YJIT now reserves a range of addresses on boot. This region start out
with no access permission at all so buggy attempts to jump to the region
crashes like before this change. To get this hardening at finer
granularity than the page size, we fill each page with trapping
instructions when we first allocate physical memory for the page.
Most of the time applications don't need 256 MiB of executable code, so
allocating on-demand ends up doing less total work than before. Case in
point, a simple `ruby --yjit-call-threshold=1 -eitself` takes about
half as long after this change. In terms of memory consumption, here is
a table to give a rough summary of the impact:
| Peak RSS in MiB | -eitself example | railsbench once |
| :-------------: | ---------------: | --------------: |
| before | 265 | 377 |
| after | 11 | 143 |
| no YJIT | 10 | 101 |
A new module is introduced to handle allocation bookkeeping.
`CodePtr` is moved into the module since it has a close relationship
with the new `VirtualMemory` struct. This new interface has a slightly
smaller surface than before in that marking a region as writable is no
longer a public operation.
The annocheck supports ELF format binaries compiled for any OS and for any
architecture. It can work in not only Linux but also FreeBSD and Solaris.
It is designed to be independent of the host OS and the architecture.
Even in Mac, as the binaries are Mach-O foramt, the annocheck fails correctly
with the message.
e.g. Test binaries compiled for Mac OSX 10.13.6 (target_os: darwin17) in Fedora 35.
```
$ cat /etc/fedora-release
Fedora release 35 (Thirty Five)
$ file ruby
ruby: Mach-O 64-bit x86_64 executable, flags:<NOUNDEFS|DYLDLINK|TWOLEVEL|WEAK_DEFINES|BINDS_TO_WEAK|PIE>
$ annocheck ruby
annocheck: Version 10.66.
annocheck: Warning: ruby: is not an ELF format file.
```
See <https://sourceware.org/bugzilla/show_bug.cgi?id=29173> for details.
... because insns_info_index could not be zero here. Also it adds an
invariant check for that.
This change will prevent the following warning of GCC 12.1
http://rubyci.s3.amazonaws.com/arch/ruby-master/log/20220613T000004Z.log.html.gz
```
compile.c:2230:39: warning: array subscript 2147483647 is outside array bounds of ‘struct iseq_insn_info_entry[2147483647]’ [-Warray-bounds]
2230 | insns_info[insns_info_index-1].line_no != adjust->line_no) {
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~
```
... by assigning a dummy value to the allocated stack.
http://rubyci.s3.amazonaws.com/arch/ruby-master/log/20220613T000004Z.log.html.gz
```
cont.c: In function ‘cont_restore_0.constprop’:
cont.c:1489:28: warning: ‘*sp’ may be used uninitialized [-Wmaybe-uninitialized]
1489 | space[0] = *sp;
| ^~~
```
Also it adds some comments about the hack of dummy stack allocation.
It sent the char to check even to non-tty, e.g., pipe.
This causes `unknown command: "\xE2\x96\xBDstart ` warnings on
ruby's parallel test on Windows, where non-standard FDs cannot be
passed to child processes.
https://github.com/ruby/reline/commit/0d373647fb
These gemspecs already work most of the times. When they are installed
normally, the require_paths in the gemspec stub line becomes actually
correct, and the incorrect value in the real gemspec is ignored. It only
becomes an issue in standalone mode.
In Ruby 3.2, `Kernel#=~` has been removed, and that means that it
becomes harder for us to gracefully deal with this error in standalone
mode, because it now happens earlier due to calling `Array#=~` for this
invalid gemspec (since require_paths is incorrectly an array of arrays).
The easiest way to fix this is to actually make this just work instead
by automatically fixing the issue when reading the packaged gemspec.
https://github.com/rubygems/rubygems/commit/d3f2fe6d26
Previously we would instantiate two different packages and extract the
specification from the package twice for each gem installed. We can
reuse the installer for this so that we just need to do it once.
https://github.com/rubygems/rubygems/commit/e454f850b1
Previously, because opt_aref and opt_aset don't push a frame, when they
would call rb_hash to determine the hash value of the key, the initial
level of recursion would incorrectly use the method id at the top of the
stack instead of "hash".
This commit replaces rb_exec_recursive_outer with
rb_exec_recursive_outer_mid, which takes an explicit method id, so that
we can make the hash calculation behave consistently.
rb_exec_recursive_outer was documented as being internal, so I believe
this should be okay to change.
df317151a5 removed the code to free
rb_hook_list_t, so repeated targeting of the same bmethod started
to leak the hook list. You can observe how the maximum memory use
scales with input size in the following script with `/usr/bin/time -v`.
```ruby
o = Object.new
o.define_singleton_method(:foo) {}
trace = TracePoint.new(:return) {}
bmethod = o.method(:foo)
ARGV.first.to_i.times { trace.enable(target:bmethod){} }
4.times {GC.start}
```
After this change the maximum doesn't grow as quickly.
To plug the leak, check whether the hook list is already allocated
when enabling the targeting TracePoint for the bmethod. This fix
also allows multiple TracePoints to target the same bmethod, similar
to other valid TracePoint targets.
Finally, free the rb_hook_list_t struct when freeing the method
definition it lives on. Freeing in the GC is a good way to avoid
lifetime problems similar to the one fixed in df31715.
[Bug #18031]
I originally added the check for
RubyVM::YJIT.trace_exit_locations_enabled? to fix errors when these
tests run without the stats feature enabled. However I forgot that this
will never be true when this test is booting, so nothing was running
when the stats feature is turned on.
Instead I've decided to make a new hash in the dump file and check if
exit locations are enabled there. If they aren't enabled we return early
to avoid checking for keys that won't exit in the dumped exit locations.
I chose to add this additional enabled check because empty exit
locations might not indicate that stats isn't enabled, it could mean the
feature is entirely broken. I do want these tests to fail if stats are
on and nothing was collected.
Followup to #5970
git
Alan Wu
Burdette Lamar
Nobuyoshi Nakada
Matt Valentine-House
Takashi Kokubun
st0012
Jun Aruga
Yusuke Endoh
Samuel Williams
David Rodríguez
Takayoshi Nishida
Jaeson Lauritzen
John Hawthorn
Jemma Issroff
Noah Gibbs
Koichi Sasada
Eileen M. Uchitelle