Граф коммитов

2270 Коммитов

Автор SHA1 Сообщение Дата
Jean Boussier b4a69351ec Move FL_SINGLETON to FL_USER1
This frees FL_USER0 on both T_MODULE and T_CLASS.

Note: prior to this, FL_SINGLETON was never set on T_MODULE,
so checking for `FL_SINGLETON` without first checking that
`FL_TYPE` was `T_CLASS` was valid. That's no longer the case.
2024-03-06 13:11:41 -05:00
Samuel Williams 9ab1fa3bf5
Hide public implementation of `rb_io`. (#9568)
Remove `struct rb_io {...}`.
2024-03-06 19:47:38 +13:00
cui fliter 226a889dc7
[DOC] fix some comments
Signed-off-by: cui fliter <imcusg@gmail.com>
2024-03-05 18:50:47 +09:00
Jean Boussier c09e5ad17d Clarify C API documentation about pinned classes
They are not only pinned, but also immortal. Even if the
constant referencing them is removed, they will remain alive.

It's a precision worth noting.
2024-03-01 08:24:16 +01:00
Nobuyoshi Nakada d4e24021d3
Revise 9ec342e07d 2024-02-26 13:12:05 +09:00
Nobuyoshi Nakada a0f7de814a
[Bug #20296] Fix the default assertion message 2024-02-26 12:29:23 +09:00
Misaki Shioi 9ec342e07d
Introduction of Happy Eyeballs Version 2 (RFC8305) in Socket.tcp (#9374)
* Introduction of Happy Eyeballs Version 2 (RFC8305) in Socket.tcp

This is an implementation of Happy Eyeballs version 2 (RFC 8305) in Socket.tcp.

[Background]
Currently, `Socket.tcp` synchronously resolves names and makes connection attempts with `Addrinfo::foreach.`
This implementation has the following two problems.

1. In name resolution, the program stops until the DNS server responds to all DNS queries.
2. In a connection attempt, while an IP address is trying to connect to the destination host and is taking time, the program stops, and other resolved IP addresses cannot try to connect.

[Proposal]
"Happy Eyeballs" ([RFC 8305](https://datatracker.ietf.org/doc/html/rfc8305)) is an algorithm to solve this kind of problem. It avoids delays to the user whenever possible and also uses IPv6 preferentially.

I implemented it into `Socket.tcp` by using `Addrinfo.getaddrinfo` in each thread spawned per address family to resolve the hostname asynchronously, and using `Socket::connect_nonblock` to try to connect with multiple addrinfo in parallel.

[Outcome]

This change eliminates a fatal defect in the following cases.

Case 1. One of the A or AAAA DNS queries does not return

---
require 'socket'

class Addrinfo
  class << self
    # Current Socket.tcp depends on foreach
    def foreach(nodename, service, family=nil, socktype=nil, protocol=nil, flags=nil, timeout: nil, &block)
      getaddrinfo(nodename, service, Socket::AF_INET6, socktype, protocol, flags, timeout: timeout)
        .concat(getaddrinfo(nodename, service, Socket::AF_INET, socktype, protocol, flags, timeout: timeout))
        .each(&block)
    end

    def getaddrinfo(_, _, family, *_)
      case family
      when Socket::AF_INET6 then sleep
      when Socket::AF_INET then [Addrinfo.tcp("127.0.0.1", 4567)]
      end
    end
  end
end

Socket.tcp("localhost", 4567)
---

Because the current `Socket.tcp` cannot resolve IPv6 names, the program stops in this case. It cannot start to connect with IPv4 address.
Though `Socket.tcp` with HEv2 can promptly start a connection attempt with IPv4 address in this case.

 Case 2. Server does not promptly return ack for syn of either IPv4 / IPv6 address family

---
require 'socket'

fork do
  socket = Socket.new(Socket::AF_INET6, :STREAM)
  socket.setsockopt(:SOCKET, :REUSEADDR, true)
  socket.bind(Socket.pack_sockaddr_in(4567, '::1'))
  sleep
  socket.listen(1)
  connection, _ = socket.accept
  connection.close
  socket.close
end

fork do
  socket = Socket.new(Socket::AF_INET, :STREAM)
  socket.setsockopt(:SOCKET, :REUSEADDR, true)
  socket.bind(Socket.pack_sockaddr_in(4567, '127.0.0.1'))
  socket.listen(1)
  connection, _ = socket.accept
  connection.close
  socket.close
end

Socket.tcp("localhost", 4567)
---

The current `Socket.tcp` tries to connect serially, so when its first name resolves an IPv6 address and initiates a connection to an IPv6 server, this server does not return an ACK, and the program stops.
Though `Socket.tcp` with HEv2 starts to connect sequentially and in parallel so a connection can be established promptly at the socket that attempted to connect to the IPv4 server.

In exchange, the performance of `Socket.tcp` with HEv2 will be degraded.

---
100.times { Socket.tcp("www.ruby-lang.org", 80) }
---

This is due to the addition of the creation of IO objects, Thread objects, etc., and calls to `IO::select` in the implementation.

* Avoid NameError of Socket::EAI_ADDRFAMILY in MinGW

* Support Windows with SO_CONNECT_TIME

* Improve performance

I have additionally implemented the following patterns:

- If the host is single-stack, name resolution is performed in the main thread. This reduces the cost of creating threads.
- If an IP address is specified, name resolution is performed in the main thread. This also reduces the cost of creating threads.
- If only one IP address is resolved, connect is executed in blocking mode. This reduces the cost of calling IO::select.

Also, I have added a fast_fallback option for users who wish not to use HE.
Here are the results of each performance test.

```ruby
require 'socket'
require 'benchmark'

HOSTNAME = "www.ruby-lang.org"
PORT = 80

ai = Addrinfo.tcp(HOSTNAME, PORT)

Benchmark.bmbm do |x|
  x.report("Domain name") do
    30.times { Socket.tcp(HOSTNAME, PORT).close }
  end

  x.report("IP Address") do
    30.times { Socket.tcp(ai.ip_address, PORT).close }
  end

  x.report("fast_fallback: false") do
    30.times { Socket.tcp(HOSTNAME, PORT, fast_fallback: false).close }
  end
end
```

```
                           user     system      total        real
Domain name            0.015567   0.032511   0.048078 (  0.325284)
IP Address             0.004458   0.014219   0.018677 (  0.284361)
fast_fallback: false   0.005869   0.021511   0.027380 (  0.321891)
````

And this is the measurement result when executed in a single stack environment.

```
                           user     system      total        real
Domain name            0.007062   0.019276   0.026338 (  1.905775)
IP Address             0.004527   0.012176   0.016703 (  3.051192)
fast_fallback: false   0.005546   0.019426   0.024972 (  1.775798)
```

The following is the result of the run on Ruby 3.3.0.

(on Dual stack environment)

```
                 user     system      total        real
Ruby 3.3.0   0.007271   0.027410   0.034681 (  0.472510)
```

(on Single stack environment)

```
                 user     system      total        real
Ruby 3.3.0  0.005353   0.018898   0.024251 (  1.774535)
```

* Do not cache `Socket.ip_address_list`

As mentioned in the comment at https://github.com/ruby/ruby/pull/9374#discussion_r1482269186, caching Socket.ip_address_list does not follow changes in network configuration.
But if we stop caching, it becomes necessary to check every time `Socket.tcp` is called whether it's a single stack or not, which could further degrade performance in the case of a dual stack.
From this, I've changed the approach so that when a domain name is passed, it doesn't check whether it's a single stack or not and resolves names in parallel each time.

The performance measurement results are as follows.

require 'socket'
require 'benchmark'

HOSTNAME = "www.ruby-lang.org"
PORT = 80

ai = Addrinfo.tcp(HOSTNAME, PORT)

Benchmark.bmbm do |x|
  x.report("Domain name") do
    30.times { Socket.tcp(HOSTNAME, PORT).close }
  end

  x.report("IP Address") do
    30.times { Socket.tcp(ai.ip_address, PORT).close }
  end

  x.report("fast_fallback: false") do
    30.times { Socket.tcp(HOSTNAME, PORT, fast_fallback: false).close }
  end
end

                           user     system      total        real
Domain name            0.004085   0.011873   0.015958 (  0.330097)
IP Address             0.000993   0.004400   0.005393 (  0.257286)
fast_fallback: false   0.001348   0.008266   0.009614 (  0.298626)

* Wait forever if fallback addresses are unresolved, unless resolv_timeout

Changed from waiting only 3 seconds for name resolution when there is no fallback address available, to waiting as long as there is no resolv_timeout.
This is in accordance with the current `Socket.tcp` specification.

* Use exact pattern to match IPv6 address format for specify address family
2024-02-26 12:14:11 +09:00
Koichi Sasada d578684989 `rb_thread_lock_native_thread()`
Introduce `rb_thread_lock_native_thread()` to allocate dedicated
native thread to the current Ruby thread for M:N threads.
This C API is similar to Go's `runtime.LockOSThread()`.

Accepted at https://github.com/ruby/dev-meeting-log/blob/master/2023/DevMeeting-2023-08-24.md
(and missed to implement on Ruby 3.3.0)
2024-02-21 15:38:29 +09:00
Nobuyoshi Nakada c77f736bc1
Win32: Fix pre-defined macros for platforms
Use `_WIN64` for word-size, `_M_AMD64` for CPU-specific feature.
2024-02-11 19:43:06 +09:00
Nobuyoshi Nakada 34581410f2
Extract `RBIMPL_VA_OPT_ARGS`
Similar to splat argument in Ruby, which be expanded to `__VA_ARGS__`
with a leading comma if any arguments given, otherwise empty.
2024-02-08 18:08:42 +09:00
Nobuyoshi Nakada d31a12a210
Optional detail info at assertion failure 2024-02-08 18:08:41 +09:00
Nobuyoshi Nakada 7b3e05c392
Do not define ABI version in statically linked objects
It is for dynamically loading, useless for statically linked objects.
2024-02-04 20:33:45 +09:00
Nobuyoshi Nakada 8531ac3115
Suppress unused-local-typedef warnings 2024-02-01 21:17:37 +09:00
KJ Tsanaktsidis 807714447e Pass down "stack start" variables from closer to the top of the stack
This commit changes how stack extents are calculated for both the main
thread and other threads. Ruby uses the address of a local variable as
part of the calculation for machine stack extents:

* pthreads uses it as a lower-bound on the start of the stack, because
  glibc (and maybe other libcs) can store its own data on the stack
  before calling into user code on thread creation.
* win32 uses it as an argument to VirtualQuery, which gets the extent of
  the memory mapping which contains the variable

However, the local being used for this is actually too low (too close to
the leaf function call) in both the main thread case and the new thread
case.

In the main thread case, we have the `INIT_STACK` macro, which is used
for pthreads to set the `native_main_thread->stack_start` value. This
value is correctly captured at the very top level of the program (in
main.c). However, this is _not_ what's used to set the execution context
machine stack (`th->ec->machine_stack.stack_start`); that gets set as
part of a call to `ruby_thread_init_stack` in `Init_BareVM`, using the
address of a local variable allocated _inside_ `Init_BareVM`. This is
too low; we need to use a local allocated closer to the top of the
program.

In the new thread case, the lolcal is allocated inside
`native_thread_init_stack`, which is, again, too low.

In both cases, this means that we might have VALUEs lying outside the
bounds of `th->ec->machine.stack_{start,end}`, which won't be marked
correctly by the GC machinery.

To fix this,

* In the main thread case: We already have `INIT_STACK` at the right
  level, so just pass that local var to `ruby_thread_init_stack`.
* In the new thread case: Allocate the local one level above the call to
  `native_thread_init_stack` in `call_thread_start_func2`.

[Bug #20001]

fix
2024-01-19 09:55:12 +11:00
KJ Tsanaktsidis 396e94666b Revert "Pass down "stack start" variables from closer to the top of the stack"
This reverts commit 4ba8f0dc99.
2024-01-12 17:58:54 +11:00
KJ Tsanaktsidis 4ba8f0dc99 Pass down "stack start" variables from closer to the top of the stack
The implementation of `native_thread_init_stack` for the various
threading models can use the address of a local variable as part of the
calculation of the machine stack extents:

* pthreads uses it as a lower-bound on the start of the stack, because
  glibc (and maybe other libcs) can store its own data on the stack
  before calling into user code on thread creation.
* win32 uses it as an argument to VirtualQuery, which gets the extent of
  the memory mapping which contains the variable

However, the local being used for this is actually allocated _inside_
the `native_thread_init_stack` frame; that means the caller might
allocate a VALUE on the stack that actually lies outside the bounds
stored in machine.stack_{start,end}.

A local variable from one level above the topmost frame that stores
VALUEs on the stack must be drilled down into the call to
`native_thread_init_stack` to be used in the calculation. This probably
doesn't _really_ matter for the win32 case (they'll be in the same
memory mapping so VirtualQuery should return the same thing), but
definitely could matter for the pthreads case.

[Bug #20001]
2024-01-12 17:29:48 +11:00
Peter Zhu 824ff48adc Move internal ST functions to internal/st.h
st_replace and st_init_existing_table_with_size are functions used
internally in Ruby and should not be publicly visible.
2023-12-25 10:41:12 -05:00
Yukihiro "Matz" Matsumoto 98eeadc932
Development of 3.4.0 started. 2023-12-25 18:13:40 +09:00
Samuel Williams 260bf60e52
Correctly release the underlying file mapping. (#9340)
* Avoiding using `Tempfile` which was retaining the file preventing it from unlinking.
2023-12-25 14:20:53 +13:00
Samuel Williams 37753f163e
IO::Buffer improvements and documentation. (#9329)
* Restore experimental warnings.

* Documentation and code structure improvements.

* Improved validation of flags, clarified documentation of argument handling.

* Remove inconsistent use of `Example:` and add example to `null?`.

* Expose `IO::Buffer#private?` and add test.
2023-12-25 02:03:36 +13:00
Alan Wu 0c05551f58 Typo fixes for public headers [ci skip] 2023-12-21 20:34:49 -05:00
John Hawthorn 1710eb9367 [DOC] Fix rb_postponed_job_register_once typo
Co-authored-by: Dustin Brown <dbrown9@gmail.com>
2023-12-21 09:17:22 -08:00
Satoshi Tagomori e51f9e9f75 rb_ext_resolve_symbol: C API to resolve and return externed symbols [Feature #20005]
This is a C API for extensions to resolve and get function symbols of other extensions.
Extensions can check the expected symbol is correctly loaded and accessible, and
use it if it is available.
Otherwise, extensions can raise their own error to guide users to setup their
environments correctly and what's missing.
2023-12-14 17:39:42 +09:00
KJ Tsanaktsidis 626daa73ec Small doc improvements for rb_postponed_job API 2023-12-13 13:35:05 +11:00
Koichi Sasada c4c39082af add `flags` to `rb_postponed_job_preregister`
for future extensions.
2023-12-10 15:39:06 +09:00
KJ Tsanaktsidis f8effa209a Change the semantics of rb_postponed_job_register
Our current implementation of rb_postponed_job_register suffers from
some safety issues that can lead to interpreter crashes (see bug #1991).
Essentially, the issue is that jobs can be called with the wrong
arguments.

We made two attempts to fix this whilst keeping the promised semantics,
but:
  * The first one involved masking/unmasking when flushing jobs, which
    was believed to be too expensive
  * The second one involved a lock-free, multi-producer, single-consumer
    ringbuffer, which was too complex

The critical insight behind this third solution is that essentially the
only user of these APIs are a) internal, or b) profiling gems.

For a), none of the usages actually require variable data; they will
work just fine with the preregistration interface.

For b), generally profiling gems only call a single callback with a
single piece of data (which is actually usually just zero) for the life
of the program. The ringbuffer is complex because it needs to support
multi-word inserts of job & data (which can't be atomic); but nobody
actually even needs that functionality, really.

So, this comit:
  * Introduces a pre-registration API for jobs, with a GVL-requiring
    rb_postponed_job_prereigster, which returns a handle which can be
    used with an async-signal-safe rb_postponed_job_trigger.
  * Deprecates rb_postponed_job_register (and re-implements it on top of
    the preregister function for compatability)
  * Moves all the internal usages of postponed job register
    pre-registration
2023-12-10 15:00:37 +09:00
KJ Tsanaktsidis aecbd66742 Add RUBY_ATOMIC_{PTR_,}FETCH macros for atomic loads
This can already be emulated by doing an atomic fetch_add of zero, but
this is more explicit.

[Bug #19994]
2023-12-10 15:00:37 +09:00
卜部昌平 51ab9ebca1 [ci skip] comment for commit be1bbd5b7d 2023-12-08 13:17:56 +09:00
Koichi Sasada 352a885a0f Thread specific storage APIs
This patch introduces thread specific storage APIs
for tools which use `rb_internal_thread_event_hook` APIs.

* `rb_internal_thread_specific_key_create()` to create a tool specific
  thread local storage key and allocate the storage if not available.
* `rb_internal_thread_specific_set()` sets a data to thread and tool
  specific storage.
* `rb_internal_thread_specific_get()` gets a data in thread and tool
  specific storage.

Note that `rb_internal_thread_specific_get|set(thread_val, key)`
can be called without GVL and safe for async signal and safe for
multi-threading (native threads). So you can call it in any internal
thread event hooks. Further more you can call it from other native
threads. Of course `thread_val` should be living while accessing the
data from this function.

Note that you should not forget to clean up the set data.
2023-12-08 13:16:19 +09:00
Nobuyoshi Nakada 0cdad3b92a Add `RUBY_REFERENCES`
Instead of `RUBY_REFERENCES_START` and `RUBY_REFERENCES_END`, so that
auto-indent works well.
2023-11-30 21:39:28 +09:00
Nobuyoshi Nakada 30f7b7a053 Prefix `REF_EDGE` and `REFS_LIST_PTR` with `RUBY_`
Also move `struct` so that `typedef`-ed names can be used.
2023-11-30 21:39:28 +09:00
Jean Boussier 23a7714343 Refactor and fix the GVL instrumentation API
This entirely changes how it is tested. Rather than to use counters
we now record the timeline of events with associated threads which
makes it much easier to assert that certains events are only preceded
by a specific event, and makes it much easier to debug unexpected
timelines.

Co-Authored-By: Étienne Barrié <etienne.barrie@gmail.com>
Co-Authored-By: JP Camara <jp@jpcamara.com>
Co-Authored-By: John Hawthorn <john@hawthorn.email>
2023-11-27 17:37:57 +01:00
Nobuyoshi Nakada e81c380c0f
Constify `RUBY_REFERENCES_START` tables 2023-11-26 16:04:23 +09:00
Jean Boussier 9ca41e9991 GVL Instrumentation: pass thread->self as part of event data
Context: https://github.com/ivoanjo/gvl-tracing/pull/4

Some hooks may want to collect data on a per thread basis.
Right now the only way to identify the concerned thread is to
use `rb_nativethread_self()` or similar, but even then because
of the thread cache or MaNy, two distinct Ruby threads may report
the same native thread id.

By passing `thread->self`, hooks can use it as a key to store
the metadata.

NB: Most hooks are executed outside the GVL, so such data collection
need to use a thread-safe data-structure, and shouldn't use the
reference in other ways from inside the hook.

They must also either pin that value or handle compaction.
2023-11-13 08:45:20 +01:00
Nobuyoshi Nakada 64f03460ba
[DOC] Update comment for `DECIMAL_SIZE_OF_BITS` 2023-11-11 19:26:13 +09:00
Jean Boussier 7efe0669ae TypedData_Make_Struct0: cast RTYPEDDATA_GET_DATA return pointer
Fixes:

```
/usr/local/ruby/include/ruby-3.3.0+0/ruby/internal/core/rtypeddata.h:467:33:
error: invalid conversion from ‘void*’ to ‘parser_t*’ [-fpermissive]
  467 |     (sval) = RTYPEDDATA_GET_DATA(result); \
      |              ~~~~~~~~~~~~~~~~~~~^~~~~~~~
      |                                 |
      |                                 void*
```
2023-11-08 11:33:01 +01:00
Peter Zhu 392238e3fd Implement embedded TypedData objects
This commit adds a new flag RUBY_TYPED_EMBEDDABLE that allows the data
of a TypedData object to be embedded after the object itself. This will
improve cache locality and allow us to save the 8 byte data pointer.

Co-Authored-By: Jean Boussier <byroot@ruby-lang.org>
2023-11-07 15:48:06 -05:00
Nobuyoshi Nakada ee90a7f981
[DOC] Update the document for `FilePathStringValue` 2023-11-02 10:31:13 +09:00
Daisuke Aritomo 4adf418be9 [Feature #10602] Add new API rb_profile_thread_frames()
Add a new API rb_profile_thread_frames(), which is essentialy a
per-thread version of rb_profile_frames().

While the original rb_profile_frames() always returns results about the
current active thread obtained by GET_EC(), this new API takes a Thread
to be profiled as an argument.

This should come in handy when profiling I/O-bound programs such as
webapps, since this new API allows us to learn about Threads performing
I/O (which do not have the GVL).

Profiling worker threads (such as Sidekiq workers) may be another
application.

Implements [Feature #10602]

Co-authored-by: Mike Perham <mike@perham.net>
2023-10-31 11:16:18 +09:00
Koichi Sasada be1bbd5b7d M:N thread scheduler for Ractors
This patch introduce M:N thread scheduler for Ractor system.

In general, M:N thread scheduler employs N native threads (OS threads)
to manage M user-level threads (Ruby threads in this case).
On the Ruby interpreter, 1 native thread is provided for 1 Ractor
and all Ruby threads are managed by the native thread.

From Ruby 1.9, the interpreter uses 1:1 thread scheduler which means
1 Ruby thread has 1 native thread. M:N scheduler change this strategy.

Because of compatibility issue (and stableness issue of the implementation)
main Ractor doesn't use M:N scheduler on default. On the other words,
threads on the main Ractor will be managed with 1:1 thread scheduler.

There are additional settings by environment variables:

`RUBY_MN_THREADS=1` enables M:N thread scheduler on the main ractor.
Note that non-main ractors use the M:N scheduler without this
configuration. With this configuration, single ractor applications
run threads on M:1 thread scheduler (green threads, user-level threads).

`RUBY_MAX_CPU=n` specifies maximum number of native threads for
M:N scheduler (default: 8).

This patch will be reverted soon if non-easy issues are found.

[Bug #19842]
2023-10-12 14:47:01 +09:00
Daisuke Aritomo 6d28f96986
[DOC] Fix description for `rb_postponed_job_register_one()`
The current documentation for `rb_postponed_job_register_one()` is
explaining the differences with itself, where it should be explaining
the differences with `rb_postponed_job_register()`.
2023-10-04 14:12:50 +09:00
Benoit Daloze 4468b6ef3c Fix documentation for rb_warn() and friends
* rb_warn() does not warn if $VERBOSE is nil, the "always" is wrong.
* Talk about $VERBOSE and not -W since $VERBOSE can be changed at runtime.
2023-09-29 12:53:40 +02:00
Nobuyoshi Nakada 50520cc193
[DOC] Missing comment markers 2023-09-27 16:18:05 +09:00
John Hawthorn 380b42fe6a Fix comment for rb_enc_str_new [ci skip] 2023-09-16 11:29:40 -07:00
Sutou Kouhei 90dad2b128 memory_view: Avoid using bit field
Bit field's memory layout is implementation-defined.

See also:
https://wiki.sei.cmu.edu/confluence/display/c/EXP11-C.+Do+not+make+assumptions+regarding+the+layout+of+structures+with+bit-fields

If memory layout is implementation-defined, it's difficult to use from
FFI library such as Ruby-FFI.
2023-09-09 07:30:04 +09:00
Jean Boussier 2d37b44603 Document that thread event hooks are called without the GVL
Except for the `RESUMED` event.
2023-09-07 18:31:16 +02:00
Samuel Williams e46e48d690
Expose `rb_process_status_wait` and hide `rb_process_status_waitv`. (#8316) 2023-08-29 22:24:55 +12:00
Samuel Williams caf48487ca
Restore `HAVE_RB_IO_T` macro for compatibility with `kgio`, `unicorn`, etc. (#8286) 2023-08-28 21:50:05 +09:00
卜部昌平 fa2712981f workaround clang-17 -Wc2x-extensions
cf: 874217f99b
2023-08-25 17:27:53 +09:00
Takashi Kokubun d405410e3c
YJIT: Move ROBJECT_OFFSET_* to yjit.c (#8157) 2023-08-02 10:15:29 -04:00