* YJIT: Allow dev_nodebug to disasm release-mode code
* Revert "YJIT: Squash canary before falling back"
This reverts commit f05ad373d8.
The stray canary issue should have been solved by
def7023ee4, alleviating this codegen
accommodation.
* s/runtime_assertions/runtime_checks/
---------
Co-authored-by: Alan Wu <XrXr@users.noreply.github.com>
This partially reverts https://github.com/ruby/ruby/pull/10944; now that
we decided to pass CFLAGS to $(CC) when assembling .S files, we don't
need these autoconf macros that capture the state of
__ARM_FEATURE{PAC|BTI}_DEFAULT.
[Bug #20601]
We already assemble our assembly files using the $(CC) compiler driver,
rather than the actual $(AS) assembler. This means that
* The C preprocessor gets run on the assembly file
* It's valid to pass gcc-style flags to it, like e.g.
-mbranch-protection or -fcf-protection
* If you do so, the relevant preprocessor macros like __CET__ get set
* If you really wanted to pass assembler flags, you would need to do
that using -Wa,... anyway
So I think it makes sense to pass "$(XCFLAGS) $(CFLAGS) $(CPPFLAGS)" to
gcc/clang/etc when assembling, rather than passing $(ASFLAGS) (since
the flags are not actually passed to `as`, but `cc`!).
The side effect of this is that if there are mitigation flags like
-fcf-protection in $CFLAGS, then the relevant macros like __CET__ will
be defined when assembling the files.
[Bug #20601]
This commit changes the external GC API to use `--with-shared-gc=DIR` at
configure time with a directory of the external GC and uses
`RUBY_GC_LIBRARY` environment variable to load the external GC at
runtime.
**What does this PR do?**
This PR tweaks the `vm_push_frame` function to add an explicit compiler
fence (`atomic_signal_fence`) to ensure profilers that use signals
to interrupt applications (stackprof, vernier, pf2, Datadog profiler)
can safely sample from the signal handler.
**Motivation:**
The `vm_push_frame` was specifically tweaked in
https://github.com/ruby/ruby/pull/3296 to initialize the a frame
before updating the `cfp` pointer.
But since there's nothing stopping the compiler from reordering
the initialization of a frame (`*cfp =`) with the update of the cfp
pointer (`ec->cfp = cfp`) we've been hesitant to rely on this on
the Datadog profiler.
In practice, after some experimentation + talking to folks, this
reordering does not seem to happen.
But since modern compilers have a way for us to exactly tell them
not to do the reordering (`atomic_signal_fence`), this seems even
better.
I've actually extracted `vm_push_frame` into the "Compiler Explorer"
website, which you can use to see the assembly output of this function
across many compilers and architectures: https://godbolt.org/z/3oxd1446K
On that link you can observe two things across many compilers:
1. The compilers are not reordering the writes
2. The barrier does not change the generated assembly output
(== has no cost in practice)
**Additional Notes:**
The checks added in `configure.ac` define two new macros:
* `HAVE_STDATOMIC_H`
* `HAVE_DECL_ATOMIC_SIGNAL_FENCE`
Since Ruby generates an arch-specific `config.h` header with
these macros upon installation, this can be used by profilers
and other libraries to test if Ruby was compiled with the fence enabled.
**How to test the change?**
As I mentioned above, you can check https://godbolt.org/z/3oxd1446K
to confirm the compiled output of `vm_push_frame` does not change
in most compilers (at least all that I've checked on that site).
`ac_abs_builddir` can be empty when the build is top-level (not
subdirs, and Ruby is usually the case). In such case, the MINIRUBY is
expanded to `$RUBY -I -r'$(arch)-fake'`, which interprets `-r$(arch)-fake`
as an argument to `-I` option. This led to:
- Not loading the fake config file
- Then not setting `CROSS_COMPILING` constant during extmk.rb execution
- Then misusing cross-compiled `./miniruby` instead of baseruby to generate
files used in exts.
This commit fixes the issue by handling the empty `ac_abs_builddir` case
properly.
This changes the automatic detection of -fstack-protector,
-D_FORTIFY_SOURCE, and -mbranch-protection to write to $hardenflags
instead of $XCFLAGS. The definition of $cflags is changed to
"$hardenflags $orig_cflags $optflags $debugflags $warnflags" to match.
Furthermore, these flags are _prepended_ to $hardenflags, rather than
appended.
The implications of doing this are as follows:
* If a CRuby builder specifies cflags="-mbranch-protection=foobar" at
the ./configure script, and the configure script detects that
-mbranch-protection=pac-ret is accepted, then GCC will be invoked as
"gcc -mbranch-protection=pac-ret -mbranch-protection=foobar". Since
the last flags take precedence, that means that user-supplied values
of these flags in $cflags will take priority.
* Likewise, if a CRuby builder explicitly specifies
"hardenflags=-mbranch-protection=foobar", because we _prepend_ to
$hardenflags in our autoconf script, we will still invoke GCC as
"gcc -mbranch-protection=pac-ret -mbranch-protection=foobar".
* If a CRuby builder specifies CFLAGS="..." at the configure line,
automatic detection of hardening flags is ignored as before.
* C extensions will _also_ be built with hardening flags now as well
(this was not the case by default before because the detected flags
went into $XCFLAGS).
Additionally, as part of this work, I changed how the detection of
PAC/BTI in Context.S works. Rather than appending the autodetected
option to ASFLAGS, we simply compile a set of test programs with the
actual CFLAGS in use to determine what PAC/BTI settings were actually
chosen by the builder. Context.S is made aware of these choices through
some custom macros.
The result of this work is that:
* Ruby will continue to choose some sensible defaults for hardening
options for the C compiler
* Distributors are able to specify CFLAGS that are consistent with their
distribution and override these defaults
* Context.S will react to whatever -mbranch-protection is actually in
use, not what was autodetected
* Extensions get built with hardening flags too.
[Bug #20154]
[Bug #20520]
It is questionable whether __builtin_setjmp should default to yes
at all, but since it appears to still have problems on this platform,
it seems safest to disable it.
Fixes [Bug #14480]
This reverts commit 8277cf0799.
This change break to build with `rbenv install ruby-dev` with the following error.
```
touch yjit/target/release/libyjit.a
transdb.h updated
./tool/darwin-ar: line 6: /nm: No such file or directory
./tool/darwin-ar: line 6: exec: /nm: cannot execute: No such file or directory
partial linking yjit/target/release/libyjit.a into yjit/target/release/libyjit.o
```
Add a Ruby script mode to `tool/missing-baseruby.bat` that checks if
`RUBY_VERSION` meets the required version. This will enable similar
checks on mswin as well.
By replacing `ALLOBJS` suffix with intermediate file suffixes instead
of roughly removing by wildcards. Made `cleanlibs` append `.dSYM`
suffix for each word in `TARGET_SO`, not the end of the entire list.
Recent `codesign` seems to emit a "replacing existing signature"
warning even if it was an adhoc signature added by the linker. Since
"adhoc" signatures do not cause a failure when replaced, it is just
unnecessary and noisy.
The `POSTLINK` variable had been used only for darwin platforms, but
it's now used for WebAssembly/WASI as well. To make shared extension
libraries properly post-processed, we need to append `POSTLINK` to
`LINK_SO` but it was done only for darwin platforms. This commit
generalizes the appending to all platforms.
We are going to add dynamic linking support for WASI platform. The
`LDSHARED` definition is used to link shared libraries for building ruby
binaries and extensions.
Before PIC era, we could assume that the stack is not unwound by
imported functions since all imported functions are WASI syscalls and
they don't use Asyncify at all. However, PIC binary can import functions
from other modules and we cannot guarantee that they won't unwind
the stack.
We need to use the same options for both C compiler and assembler
when `-mbranch-protection` is guessed by configure. Otherwise,
`coroutine/arm64/Context.{h,S}` will use incompatible PAC strategies.
* Allows macOS users to use M:N threads (and technically FreeBSD, though it has not been verified on FreeBSD)
* Include sys/event.h header check for macros, and include sys/event.h when present
* Rename epoll_fd to more generic kq_fd (Kernel event Queue) for use by both epoll and kqueue
* MAP_STACK is not available on macOS so conditionall apply it to mmap flags
* Set fd to close on exec
* Log debug messages specific to kqueue and epoll on creation
* close_invalidate raises an error for the kqueue fd on child process fork. It's unclear rn if that's a bug, or if it's kqueue specific behavior
Use kq with rb_thread_wait_for_single_fd
* Only platforms with `USE_POLL` (linux) had changes applied to take advantage of kernel event queues. It needed to be applied to the `select` so that kqueue could be properly applied
* Clean up kqueue specific code and make sure only flags that were actually set are removed (or an error is raised)
* Also handle kevent specific errnos, since most don't apply from epoll to kqueue
* Use the more platform standard close-on-exec approach of `fcntl` and `FD_CLOEXEC`. The io-event gem uses `ioctl`, but fcntl seems to be the recommended choice. It is also what Go, Bun, and Libuv use
* We're making changes in this file anyways - may as well fix a couple spelling mistakes while here
Make sure FD_CLOEXEC carries over in dup
* Otherwise the kqueue descriptor should have FD_CLOEXEC, but doesn't and fails in assert_close_on_exec
Previously, the embedded semicolon in BASERUBY if BASERUBY is
not available breaks tarball builds without BASERUBY when using
OpenBSD make, due to the inability to escape MFLAGS correctly.
This moves the same BASERUBY code into a separate file, avoiding
the MFLAGS quoting issue.
BASERUBY must be passed to build-ext because it is required
by ripper since the introduction of lrama.
Fixes [Bug #19683]
Co-authored-by: Nobuyoshi Nakada <nobu@ruby-lang.org>
* [win32] fix compilation for windows-arm64
Credits to MSYS2 Ruby package using this patch.
* [win32] nm use full options
Fix compilation error when using MSYS2 environment.
Credits to MSYS2 Ruby package using this patch.
* [win32] detect llvm-windres (used for windows-arm64)
When adding preprocessor option for llvm-windres (using clang as
parameter), it fails. Thus, do not add this.
It's needed to be able to compile windows-arm64 version, because MSYS2
toolchain is LLVM based (instead of GCC/binutils).
* [win32] pioinfo detection for windows-arm64
This fixes "unexpected ucrtbase.dll" for native windows-arm64 ruby
binary. It does not solve issue with x64 version emulated on this
platform.
Value of pioinfo pointer can be found in ucrtbase.dll at latest adrp/add
sequence before return of _isatty function. This works for both release
and debug ucrt.
Due to the nature of aarch64 ISA (vs x86 or x64), it's needed to
disassemble instructions to retrieve offset value, which is a bit more
complicated than matching specific string patterns.
Details about adrp/add usage can be found in this blog post:
https://devblogs.microsoft.com/oldnewthing/20220809-00/?p=106955
For instruction decoding, the Arm documentation was used as a reference.
Takashi Kokubun
Nobuyoshi Nakada
Kazuhiro NISHIYAMA
KJ Tsanaktsidis
Peter Zhu
Ivo Anjo
Yuta Saito
Yusuke Endoh
ydah
Jeremy Evans
Hiroshi SHIBATA
卜部昌平
Matt Valentine-House
JP Camara
Pierrick Bouvier