We can't always designate a timer thread, so any sleepers must
also perform ubf wakeups. Note: a similar change needs to be
made for rb_thread_fd_select and rb_wait_for_single_fd.
[ruby-core:88088] [Misc #14937] [Bug #5343]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64111 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
thread_pthread.c relies on ppoll for rb_sigwait_sleep, so ensure
the compatibility wrapper is available for it.
[Bug #14950]
Reported-by: SHIBATA Hiroshi <hsbt@ruby-lang.org>
Reported-by: Greg L <Greg.mpls@gmail.com>
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64110 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
thread_pthread.c relies on ppoll for rb_sigwait_sleep, so ensure
the compatibility wrapper is available for it.
Reported-by: SHIBATA Hiroshi <hsbt@ruby-lang.org>
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64109 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
We still need to to designate a timer thread after registering target
thread for the ubf list.
Oops :x
Note: I was never able to reproduce
test/thread/test_queue.rb::test_thr_kill failures on my on
Debian machines.
[ruby-core:88088] [Misc #14937] [Bug #5343]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64108 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
To reduce resource use and reduce CI failure; remove
timer-thread. Single-threaded Ruby processes (including forked
children) will never see extra thread overhead. This prevents
glibc and jemalloc from going into multi-threaded mode and
initializing locks or causing fragmentation via arena explosion.
The GVL is implements its own wait-queue as a ccan/list to
permit controlling wakeup order. Timeslice under contention is
handled by a designated timer thread (similar to choosing a
"patrol_thread" for current deadlock checking).
There is only one self-pipe, now, as wakeups for timeslice are
done independently using condition variables. This reduces FD
pressure slightly.
Signal handling is handled directly by a Ruby Thread (instead
of timer-thread) by exposing signal self-pipe to callers of
rb_thread_fd_select, native_sleep, rb_wait_for_single_fd, etc...
Acquiring, using, and releasing the self-pipe is exposed via 4
new internal functions:
1) rb_sigwait_fd_get - exclusively acquire timer_thread_pipe.normal[0]
2) rb_sigwait_fd_sleep - sleep and wait for signal (and no other FDs)
3) rb_sigwait_fd_put - release acquired result from rb_sigwait_fd_get
4) rb_sigwait_fd_migrate - migrate signal handling to another thread
after calling rb_sigwait_fd_put.
rb_sigwait_fd_migrate is necessary for waitpid callers because
only one thread can wait on self-pipe at a time, otherwise a
deadlock will occur if threads fight over the self-pipe.
TRAP_INTERRUPT_MASK is now set for the main thread directly in
signal handler via rb_thread_wakeup_timer_thread.
Originally, I wanted to use POSIX timers
(timer_create/timer_settime) for this. Unfortunately, this
proved unfeasible as Mutex#sleep resumes on spurious wakeups and
test/thread/test_cv.rb::test_condvar_timed_wait failed. Using
pthread_sigmask to mask out SIGVTALRM fixed that test, but
test/fiddle/test_function.rb::test_nogvl_poll proved there'd be
some unavoidable (and frequent) incompatibilities from that
approach.
Finally, this allows us to drop thread_destruct_lock and
interrupt current ec directly.
We don't need to rely on vm->thread_destruct_lock or a coherent
vm->running_thread on any platform. Separate timer-thread for
time slice and signal handling is relegated to thread_win32.c,
now.
[ruby-core:88088] [Misc #14937]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64107 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
This should fix CI failure under MJIT, 100ms wait may not be
enough for signal handler to fire:
http://ci.rvm.jp/results/trunk-mjit@silicon-docker/1169472
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64105 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Otherwise, an altstack may live past ObjectSpace destruction
and xfree-ing the altstack will segfault.
[ruby-core:85621] [Feature #14487]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64102 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* ext/openssl/extconf.rb: LibreSSL headers emit "overriding WinCrypt
defines" warnings if wincrypt.h has been included (except for
x509.h) on Windows. get rid of including the header by defining
NOCRYPT macro.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64101 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
to use compaction with --jit-save-temps.
Prior to this commit, JIT compaction didn't work with --jit-save-temps
but it wasn't intentional.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64100 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
for the ease of MSP-Greg's CI. I'll track this failure as Bug#14948 and
fix it later.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64099 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
When --jit-wait is specified, `unit_queue.length` is always 0 and it's
not a good metric.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64097 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Repack rb_thread_struct, rb_execution_context_struct, args_info and
iseq_compile_data to save 1 word per struct.
re_pattern_buffer remains unpacked due to the possible binary
compatibility.
[Fix GH-1907]
Based on the patch
From: Lourens Naudé <lourens@bearmetal.eu>
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64096 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
When all compilation finishes or the number of JIT-ed code reaches
--jit-max-cache, this compacts all generated code to a single .so file
and re-loads all methods from it.
In the future, it may trigger compaction more frequently and/or limit
the maximum times of compaction to prevent unlimited memory usage.
So the current behavior is experimental, but at least the performance
improvement in this commit won't be removed.
=== Benchmark ===
In this benchmark, I'll compare following four conditions:
* trunk: r64082
* trunk JIT: r64082 w/ --jit
* single-so JIT: This commit w/ --jit
* objfcn JIT: This branch https://github.com/k0kubun/ruby/tree/objfcn w/ --jit,
which is shinh's objfcn https://github.com/shinh/ruby/tree/objfcn rebased from this commit
```
$ uname -a
Linux bionic 4.15.0-29-generic #31-Ubuntu SMP Tue Jul 17 15:39:52 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
```
* Micro benchmark
Using this script https://gist.github.com/k0kubun/10e6d3387c9ab1b134622b2c9d76ef51,
calls some amount of different methods that just return `nil`. The following tables
are its average duration seconds of 3 measurements.
Smaller is better.
** 1 method (seconds)
| | trunk | trunk JIT | single-so JIT | objfcn JIT |
|:------|:------------------|:------------------|:------------------|:------------------|
| Time | 5.576067774333296 | 5.915551971666446 | 5.833641665666619 | 5.845915191666639 |
| Ratio | 1.00x | 1.06x | 1.05x | 1.05x |
** 50 methods (seconds)
| | trunk | trunk JIT | single-so JIT | objfcn JIT |
|:------|:------------------|:------------------|:------------------|:------------------|
| Time | 3.1661167996666677| 6.125825928333342 | 4.135432743666665 | 3.750358728333348 |
| Ratio | 1.00x | 1.93x | 1.31x | 1.18x |
** 1500 methods (seconds)
| | trunk | trunk JIT | single-so JIT | objfcn JIT |
|:------|:------------------|:------------------|:------------------|:------------------|
| Time | 5.971650823666664 | 19.579182102999994| 10.511108153999961| 10.854653588999932|
| Ratio | 1.00x | 3.28x | 1.76x | 1.82x |
* Discourse
Using the same benchmark strategy as https://bugs.ruby-lang.org/issues/14490 with
this branch https://github.com/k0kubun/discourse/commits/benchmark2 forked from discourse
v1.8.11 to support running trunk.
1. Run ruby script/bench.rb to warm up profiling database
2. Run RUBYOPT='--jit-verbose=1 --jit-max-cache=10000' RAILS_ENV=profile bin/puma -e production
3. WAIT 5-15 or so minutes for all jitting to stop so we have no cross talk
4. Run ab -n 100 http://localhost:9292/
5. Wait for all new jitting to finish
6. Run ab -n 100 http://localhost:9292/
** Response time (ms)
Here is the response time milliseconds for each percentile.
Skipping 99%ile because it's the same as 100%ile in 100 calls.
| | trunk| trunk|single|objfcn|
| | | JIT|so JIT| JIT|
|:----|:-----|:-----|:-----|:-----|
| 50% | 38 | 45 | 41 | 43 |
| 66% | 39 | 50 | 44 | 44 |
| 75% | 47 | 51 | 46 | 45 |
| 80% | 49 | 52 | 47 | 47 |
| 90% | 50 | 63 | 50 | 52 |
| 95% | 60 | 79 | 52 | 55 |
| 98% | 91 | 114 | 91 | 91 |
|100% | 97 | 133 | 96 | 99 |
** Ratio (smaller is better)
Here is the response time increase ratio against no-JIT trunk's one.
| | trunk| trunk|single|objfcn|
| | | JIT|so JIT| JIT|
|:----|:-----|:-----|:-----|:-----|
| 50% | 1.00x| 1.18x| 1.08x| 1.13x|
| 66% | 1.00x| 1.28x| 1.13x| 1.13x|
| 75% | 1.00x| 1.09x| 0.98x| 0.96x|
| 80% | 1.00x| 1.06x| 0.96x| 0.96x|
| 90% | 1.00x| 1.26x| 1.00x| 1.04x|
| 95% | 1.00x| 1.32x| 0.87x| 0.92x|
| 98% | 1.00x| 1.25x| 1.00x| 1.00x|
|100% | 1.00x| 1.37x| 0.99x| 1.02x|
While 50 and 60 %ile are still worse than no-JIT trunk, 75, 80, 90, 95,
98 and 100% are not slower than that.
So now it's a little harder to say "MJIT slows down Rails applications".
Probably I can close [Bug #14490] now. Let's start improving it.
Close https://github.com/ruby/ruby/pull/1921
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64094 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* win32/win32.c (constat_parse): split long buffer and limit write
size on a console, as well as rb_w32_write.
[ruby-dev:50597] [Bug #14942]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64092 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* lib/unicode_normalize/normalize.rb: Fix the range check for trailing
Hangul jamo characters in Unicode normalization. Different from
leading or vowel jamos, where LBASE and VBASE are actual characters,
a value equal to TBASE expresses the absence of a trailing jamo.
This fix is technically correct, but there was no bug because
the regular expressions in lib/unicode_normalize/tables.rb
eliminate jamos equal to TBASE from normalization processing.
* test/test_unicode_normalize.rb: Add preventive test
test_no_trailing_jamo based on
d134809cd3
just for the case we ever get a regression.
This closes issue #14934, thanks to MaLin (Lin Ma) for reporting.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64087 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* test/test_unicode_normalize.rb: extract tests that do not depend
on NormalizationTest.txt data file from conditionally constructed
part of TestUnicodeNormalize class, to always run them even if
the data file isn't found.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64086 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* test/test_unicode_normalize.rb: search ucd directory
first if it exists. This change follows r61415.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64085 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
If we keep ubf set after unregistering, there is a window for
other threads (including timer thread) to put this thread back
on the ubf_list right away. Entering ubf_list unexpectedly
after GVL acquisition may cause spurious wakeup and trigger
unexpected behavior.
Finally, clear ubf before acquiring GVL, to since ubf is useless
during GVL acquisition anyways and we don't want to waste cycles
in other threads calling ubf for useless work.
[ruby-core:88141] [Bug #14945]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64083 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Both methods Hash#length and Hash#size share the same source code in
Ruby, but they also share the same documentation. Now when you look at
the documentation of Hash#size you only see examples for Hash#length,
which is confusing. This commit includes Hash#size in the examples and
also remarks that both methods are equivalent to each other.
Co-authored-by: Alberto Almagro <alberto.almagro@rakuten.com>
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64081 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Just refactoring. Despite its name, the function does NOT return a
boolean but raises an exception when the class given is frozen.
I don't think the new name "rb_class_modify_check" is the best, but
it follows the precedeint "rb_ary_modify_check", and is definitely
better than "*_p".
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64078 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
and lazily delete them on termination.
This will be needed to create a large so file later.
The large number of .o files will be probably compacted before the large so
file is created.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64077 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
units, renaming it from NOT_COMPILABLE_JIT_ISEQ_FUNC.
NOT_READY_JIT_ISEQ_FUNC is for ones being compiled, so
mjit_get_iseq_func treats it specially and it shouldn't be used for the
purpose.
I renamed it instead of adding a new one because I'm not sure about the
impact for the performance by increasing the switch branches in mjit_exec.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64076 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* ruby.c (external_str_new_cstr): strings come from the external
should be tainted. [ruby-dev:50596] [Bug #14941]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64071 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Patch by yuuji.yaginuma [Fix GH-1890]
Since `remove_method` is public.
Ref: https://bugs.ruby-lang.org/issues/14133
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64069 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* dir.c (do_stat, do_lstat, do_opendir): need the length of the base
path for fstatat() when fd is valid.
* dir.c (glob_helper): fix for platforms where DT_UNKNOWN is not
available, e.g. Solaris.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64067 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* process.c (before_fork_ruby, after_fork_ruby): used only if fork()
or daemon() is available.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64065 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
normal
shyouhei
svn
nobu
k0kubun
kazu
duerst
tenderlove
mame
marcandre