Context: https://github.com/ivoanjo/gvl-tracing/pull/4
Some hooks may want to collect data on a per thread basis.
Right now the only way to identify the concerned thread is to
use `rb_nativethread_self()` or similar, but even then because
of the thread cache or MaNy, two distinct Ruby threads may report
the same native thread id.
By passing `thread->self`, hooks can use it as a key to store
the metadata.
NB: Most hooks are executed outside the GVL, so such data collection
need to use a thread-safe data-structure, and shouldn't use the
reference in other ways from inside the hook.
They must also either pin that value or handle compaction.
This commit adds a new flag RUBY_TYPED_EMBEDDABLE that allows the data
of a TypedData object to be embedded after the object itself. This will
improve cache locality and allow us to save the 8 byte data pointer.
Co-Authored-By: Jean Boussier <byroot@ruby-lang.org>
Add a new API rb_profile_thread_frames(), which is essentialy a
per-thread version of rb_profile_frames().
While the original rb_profile_frames() always returns results about the
current active thread obtained by GET_EC(), this new API takes a Thread
to be profiled as an argument.
This should come in handy when profiling I/O-bound programs such as
webapps, since this new API allows us to learn about Threads performing
I/O (which do not have the GVL).
Profiling worker threads (such as Sidekiq workers) may be another
application.
Implements [Feature #10602]
Co-authored-by: Mike Perham <mike@perham.net>
This patch introduce M:N thread scheduler for Ractor system.
In general, M:N thread scheduler employs N native threads (OS threads)
to manage M user-level threads (Ruby threads in this case).
On the Ruby interpreter, 1 native thread is provided for 1 Ractor
and all Ruby threads are managed by the native thread.
From Ruby 1.9, the interpreter uses 1:1 thread scheduler which means
1 Ruby thread has 1 native thread. M:N scheduler change this strategy.
Because of compatibility issue (and stableness issue of the implementation)
main Ractor doesn't use M:N scheduler on default. On the other words,
threads on the main Ractor will be managed with 1:1 thread scheduler.
There are additional settings by environment variables:
`RUBY_MN_THREADS=1` enables M:N thread scheduler on the main ractor.
Note that non-main ractors use the M:N scheduler without this
configuration. With this configuration, single ractor applications
run threads on M:1 thread scheduler (green threads, user-level threads).
`RUBY_MAX_CPU=n` specifies maximum number of native threads for
M:N scheduler (default: 8).
This patch will be reverted soon if non-easy issues are found.
[Bug #19842]
The current documentation for `rb_postponed_job_register_one()` is
explaining the differences with itself, where it should be explaining
the differences with `rb_postponed_job_register()`.
Existing strscan releases rely on this C API. It means that the current
Ruby master doesn't work if your Gemfile.lock has strscan unless it's
locked to 3.0.7, which is not released yet.
To fix it, let's not remove the C API we've exposed to users.
rb_reg_onig_match performs preparation, error handling, and cleanup for
matching a regex against a string. This reduces repetitive code and
removes the need for StringScanner to access internal data of regex.
Closes [Feature #19729]
Previously 2 bits of the flags on each RVALUE are reserved to store the
number of GC cycles that each object has survived. This commit
introduces a new bit array on the heap page, called age_bits, to store
that information instead.
This patch still reserves one of the age bits in the flags (the old
FL_PROMOTED0 bit, now renamed FL_PROMOTED).
This is set to 0 for young objects and 1 for old objects, and is used as
a performance optimisation for the write barrier. Fetching the age_bits
from the heap page and doing the required math to calculate if the
object was old or not would slow down the write barrier. So we keep this
bit synced in the flags for fast access.
st_copy allocates a st_table, which is not needed for hashes since it is
allocated by VWA and embedded, so this causes a memory leak.
The following script demonstrates the issue:
```ruby
20.times do
100_000.times do
{a: 1, b: 2, c: 3, d: 4, e: 5, f: 6, g: 7, h: 8, i: 9}
end
puts `ps -o rss= -p #{$$}`
end
```
* Make TAINT and UNTRUSTED flags zero
These flags do nothing already, and should break nothing.
* Remove TAINT and UNTRUSTED macros same as functions
These macros had been defined to use with `#ifdef`, but should not be
used anymore.
Since no type information is available for variadic arguments, 0 is
passed as `int` without promoting to pointer. On platforms where
`sizeof(int) < sizeof(void*)`, the terminator argument may be read
together with an adjoining word, and may not be found.
* Unify length field for embedded and heap strings
The length field is of the same type and position in RString for both
embedded and heap allocated strings, so we can unify it.
* Remove RSTRING_EMBED_LEN
* Add rb_io_path and rb_io_open_descriptor.
* Use rb_io_open_descriptor to create PTY objects
* Rename FMODE_PREP -> FMODE_EXTERNAL and expose it
FMODE_PREP I believe refers to the concept of a "pre-prepared" file, but
FMODE_EXTERNAL is clearer about what the file descriptor represents and
aligns with language in the IO::Buffer module.
* Ensure that rb_io_open_descriptor closes the FD if it fails
If FMODE_EXTERNAL is not set, then it's guaranteed that Ruby will be
responsible for closing your file, eventually, if you pass it to
rb_io_open_descriptor, even if it raises an exception.
* Rename IS_EXTERNAL_FD -> RUBY_IO_EXTERNAL_P
* Expose `rb_io_closed_p`.
* Add `rb_io_mode` to get IO mode.
---------
Co-authored-by: KJ Tsanaktsidis <ktsanaktsidis@zendesk.com>
* Documentation consistency.
* Improve consistency of `pread`/`pwrite` implementation when given length.
* Remove HAVE_PREAD / HAVE_PWRITE - it is no longer optional.
On Win32, currently, rb_nativethread_cond_t is an incomplete type
because it's a typedef for `struct rb_thread_cond_struct`. That means
you can't actually allocate a rb_nativethread_cond_t unless you also
include THREAD_IMPL_H (since its defined in thread_win32.h)
(alternatively, including vm_core.h also works).
Move the definition of rb_thread_cond_struct into thread_native.h to
alleviate this.
* Refactor Regexp#match cache implementation
Improved variable and function names
Fixed [Bug 19537] (Maybe fixed in https://github.com/ruby/ruby/pull/7694)
* Add a comment of the glossary for "match cache"
* Skip to reset match cache when no cache point on null check
Followup: ac123f167a
RB_WARN_CATEGORY_ALL_BITS is exposed in a public header, so it
makes sense for it to be updated to contain all valid bits.
Instead we introduce RB_WARN_CATEGORY_DEFAULT_BITS to list the
categories that are enabled by default.
[Feature #19538]
This new `peformance` warning category is disabled by default.
It needs to be specifically enabled via `-W:performance` or `Warning[:performance] = true`
Remove !USE_RVARGC code
[Feature #19579]
The Variable Width Allocation feature was turned on by default in Ruby
3.2. Since then, we haven't received bug reports or backports to the
non-Variable Width Allocation code paths, so we assume that nobody is
using it. We also don't plan on maintaining the non-Variable Width
Allocation code, so we are going to remove it.
st tables will maintain insertion order so we can marshal dump / load
objects with instance variables in the same order they were set on that
particular instance
[ruby-core:112926] [Bug #19535]
Co-Authored-By: Jemma Issroff <jemmaissroff@gmail.com>
When using rb_data_type_struct to wrap a C struct, that C struct can
contain VALUE references to other Ruby objects.
If this is the case then one must also define dmark and optionally
dcompact callbacks in order to allow these objects to be correctly
handled by the GC. This is suboptimal as it requires GC related logic to
be implemented by extension developers. This can be a cause of subtle
bugs when references are not marked of updated correctly inside these
callbacks.
This commit provides an alternative approach, useful in the simple case
where the C struct contains VALUE members (ie. there isn't any
conditional logic, or data structure manipulation required to traverse
these references).
In this case references can be defined using a declarative syntax
as a list of edges (or, pointers to references).
A flag can be set on the rb_data_type_struct to notify the GC that
declarative references are being used, and a list of those references
can be assigned to the dmark pointer instead of a function callback, on
the rb_data_type_struct.
Macros are also provided for simple declaration of the reference list,
and building edges.
To avoid having to also find space in the struct to define a length for
the references list, I've chosed to always terminate the references list
with RUBY_REF_END - defined as UINTPTR_MAX. My assumption is that no
single struct will ever be large enough that UINTPTR_MAX is actually a
valid reference.
This macro is broken when set to anything other than 0. And has had a
comment saying that it's broken for 3 years.
This commit deletes it and the associated logging code. It's clearly
not being used.
Co-Authored-By: Peter Zhu <peter@peterzhu.ca>
A few extension libraries, to hide all symbols except for necessary to
load, hardcode the symbols to be exported in symbol list files for
linker without even checking by `have_func`. As a workaround for such
libraries, retain `ruby_abi_version` symbol always even in released
versions for now.
The following new debug context APIs are for implementing debugger's
`next` (step over) and similar functionality.
* `rb_debug_inspector_frame_depth(dc, index)` returns `index`-th
frame's depth.
* `rb_debug_inspector_current_depth()` returns current frame depth.
The frame depth is not related to the frame index because debug
context API skips some special frames but proposed `_depth()` APIs
returns the count of all frames (raw depth).
Since https://github.com/ruby/ruby/pull/6599, RUBY_IMMEDIATE_MASK also
overlaps RUBY_Qnil. Now RB_SPECIAL_CONST_P seems confusing since both
RB_IMMEDIATE_P and RB_TEST check for RUBY_Qnil while we only need to
check RUBY_Qnil besides RUBY_IMMEDIATE_MASK. I'd like to make this
change to make it less confusing.
I confirmed that this doesn't change the number of instructions used for
the RUBY_Qfalse check on Linux x86_64 GCC and macOS arm64 Clang.
* Windows: Fix warning about undefined if_indextoname()
* Windows: Fix UNIXSocket on MINGW and make .pair more reliable
* Windows: Use nonblock=true for read tests with scheduler
* Windows: Move socket detection from File.socket? to File.stat
Add S_IFSOCK to Windows and interpret reparse points accordingly.
Enable tests that work now.
* Windows: Use wide-char functions to UNIXSocket
This fixes behaviour with non-ASCII characters.
It also fixes deletion of temporary UNIXSocket.pair files.
* Windows: Add UNIXSocket tests for specifics of Windows impl.
* Windows: fix VC build due to missing _snwprintf
Avoid usage of _snwprintf, since it fails linking ruby.dll like so:
linking shared-library x64-vcruntime140-ruby320.dll
x64-vcruntime140-ruby320.def : error LNK2001: unresolved external symbol snwprintf
x64-vcruntime140-ruby320.def : error LNK2001: unresolved external symbol vsnwprintf_l
whereas linking miniruby.exe succeeds.
This patch uses snprintf on the UTF-8 string instead.
Also remove branch GetWindowsDirectoryW, since it doesn't work.
* Windows: Fix dangling symlink test failures
Co-authored-by: Lars Kanis <kanis@comcard.de>
* Windows: Use readlink emulation for File.readlink
This fixes readlink emulation for the ERROR_MORE_DATA case and general error reporting.
It now releases GVL while readlink IO operation.
The dedicated rb_readlink was introduced in commit 2ffb87995a
in order to improve encoding and buffer allocation.
However the encoding issues are solved since ruby-3.0 switched to UTF-8
and the buffer allocation will be improved in a later commit.
* Windows: Increase the default buffer size for reparse point info
So far nearly all queries of reparse points needed two attempts to get enough buffer.
* Windows: Remove declaration of rb_w32_wreadlink
It was removed in commit 2f6fdd3aeb
Since object shapes store the capacity of an object, we no longer
need the numiv field on RObjects. This gives us one extra slot which
we can use to give embedded objects one more instance variable (for a
total of 3 ivs). This commit removes the concept of numiv from RObject.
This commit adds a `capacity` field to shapes, and adds shape
transitions whenever an object's capacity changes. Objects which are
allocated out of a bigger size pool will also make a transition from the
root shape to the shape with the correct capacity for their size pool
when they are allocated.
This commit will allow us to remove numiv from objects completely, and
will also mean we can guarantee that if two objects share shapes, their
IVs are in the same positions (an embedded and extended object cannot
share shapes). This will enable us to implement ivar sets in YJIT using
object shapes.
Co-Authored-By: Aaron Patterson <tenderlove@ruby-lang.org>
A code pattern `p + enclen(enc, p, pend)` may lead to a buffer overrun
if incomplete bytes of a UTF-8 character is placed at the end of a
string. Because this pattern is used in several places in onigmo,
this change fixes the issue in the side of `enclen`: the function should
not return a number that is larger than `pend - p`.
Co-Authored-By: Nobuyoshi Nakada <nobu@ruby-lang.org>
Jean Boussier
Nobuyoshi Nakada
Peter Zhu
Daisuke Aritomo
Koichi Sasada
Daisuke Aritomo
Benoit Daloze
John Hawthorn
Sutou Kouhei
Samuel Williams
卜部昌平
Takashi Kokubun
Kunshan Wang
Matt Valentine-House
Peter Arato
NARUSE, Yui
KJ Tsanaktsidis
TSUYUSATO Kitsune
Aaron Patterson
Sébastien Helleu
Yukihiro "Matz" Matsumoto
Jimmy Bourassa
Lars Kanis
Lars Kanis
Jemma Issroff
Yusuke Endoh
Nick Hengeveld