Граф коммитов

909 Коммитов

Автор SHA1 Сообщение Дата
mame 6c9a705032 Remove tracecoverage instructions
The instructions were used only for branch coverage.
Instead, it now uses a trace framework [Feature #14104].

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65225 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-10-20 10:45:48 +00:00
ko1 b710785f1a add disabling MJIT features option.
* configure.ac: introduce new configure option `--enable-mjit` and
  `--disable-mjit`. Default is "enable".
  `--disable-mjit` disables all of MJIT features so that `ruby --jit`
  can't enable MJIT.
  This option affect a macro `USE_MJIT`.
  This change remove `--enable/disable-install-mjit-header` option.

* Makefile.in: introduce the `ENABLE_MJIT` variable.

* common.mk: use `ENABLE_MJIT` option.

* internal.h: respect `USE_MJIT`. Same as other *.c, *.h.

* test/ruby/test_jit.rb: check `ENABLE_MJIT` key of rbconfg.rb.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65204 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-10-20 06:53:00 +00:00
mame 567ec93ee6 thread.c (rb_clear_coverages): defined out of #ifdef
It failed to build on windows.
https://ci.appveyor.com/project/ruby/ruby/builds/19655876

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65200 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-10-20 05:44:14 +00:00
mame 47ea999b46 ext/coverage/: add the oneshot mode
This patch introduces "oneshot_lines" mode for `Coverage.start`, which
checks "whether each line was executed at least once or not", instead of
"how many times each line was executed".  A hook for each line is fired
at most once, and after it is fired, the hook flag was removed; it runs
with zero overhead.

See [Feature #15022] in detail.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65195 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-10-20 05:33:04 +00:00
shyouhei 5c16b21f44 PRI_64_PREFIX "u" is called PRIu64
See also ISO9899:1999 section 7.8.1 paragraph 3.

The #ifndef is for C compilers older than C99.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64740 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-09-14 08:27:44 +00:00
nobu 38122275f3 thread.c: fix -Wformat warning where uint64_t is not unsigned long
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64731 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-09-13 10:59:12 +00:00
k0kubun 8bc5c172f9 thread.c: fix -Wformat warning
../thread.c:1219:18: warning: format specifies type 'unsigned long' but the argument has type 'rb_hrtime_t'
      (aka 'unsigned long long') [-Wformat]
                 end, now);
                 ^~~
../thread.c:1219:23: warning: format specifies type 'unsigned long' but the argument has type 'rb_hrtime_t'
      (aka 'unsigned long long') [-Wformat]
                 end, now);
                      ^~~

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64712 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-09-13 06:49:49 +00:00
normal 8a7b4b397f thread.c (rb_thread_atfork): reinitialize current th->interrupt_lock
Another thread may be holding th->interrupt_lock while our
current thread calls fork.  Therefore we must reinitialize our
own th->interrupt_lock in the child process because the owner
of the lock is only in the parent.  The original parent process
is unaffected.  We cannot destroy the lock while it has an unknown
state, either, so some implementations can leak a small amount
of memory, here (NPTL won't).

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64707 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-09-13 01:43:03 +00:00
nobu f419aa7bdf thread.c: get rid of false warnings by gcc-8
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64665 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-09-09 12:42:48 +00:00
normal ac543abe91 cont.c: set th->root_fiber to current fiber at fork
Otherwise, th->root_fiber can point to an invalid Fiber,
because Fibers do not live across fork.  So consider
whatever Fiber is running the root fiber.

[ruby-core:88723] [Bug #15041]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64589 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-08-29 08:04:09 +00:00
normal 9d4027b503 process.c: simplify SIGCHLD-based waitpid
Introduce a new rb_thread_sleep_interruptible that does not
execute interrupts before sleeping.  Skipping the interrupt
check before sleep is required for out-of-GVL ruby_waitpid_all
to function properly when setting waitpid_state.ret

Now that ubf_select can be called by the gvl.timer thread
without recursive locking gvl.lock, we can safely use
rb_threadptr_interrupt to deal with waking up sleeping
processes,

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64576 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-08-27 17:17:13 +00:00
normal 49a55f28ab thread.c: check interrupts explicitly in select/ppoll blocking regions
The logic around blocking_region_begin is confusing to me,
but the goal of this patch is to ensure rb_sigwait_fd_get
and rb_sigwait_fd_put are matched.

In other words, we don't want a thread to hold sigwait_fd
forever if an exception is raised while calling select()
or ppoll().

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64550 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-08-27 05:32:18 +00:00
normal b4084d7c36 thread.c: quiet down -Wmaybe-uninitialized on gcc 7.[2-3]
Haven't tested gcc 8, yet; but gcc 6 seems fine....

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64536 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-08-25 21:33:55 +00:00
normal 95abea43fe hrtime.h: add documentation
I updated the patch with documentation but forgot about it,
earlier :x

[ruby-core:88616] [Misc #15014]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64535 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-08-25 09:02:50 +00:00
normal b0253a7569 thread.c: use rb_hrtime_t scalar for high-resolution time operations
Relying on "struct timespec" was too annoying API-wise and
used more stack space.  "double" was a bit wacky w.r.t rounding
in the past, so now we'll switch to using a 64-bit type.

Unsigned 64-bit integer is able to give us over nearly 585
years of range with nanoseconds.  This range is good enough
for the Linux kernel internal time representation, so it
ought to be good enough for us.

This reduces the stack usage of functions while GVL is held
(and thus subject to marking) on x86-64 Linux (with ppoll):

	rb_wait_for_single_fd    120 => 104
	do_select                120 => 88

[ruby-core:88582] [Misc #15014]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64533 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-08-25 06:58:35 +00:00
normal 0a77fd04b7 thread_pthread.c: use eventfd instead of pipe on Linux
Based on r64478, any regular user creating more than 1024 pipes
on Linux will end up with tiny pipes with only a single page
capacity.  So avoid wasting user resources and use lighter
eventfd on Linux.

[ruby-core:88563] [Misc #15011]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64527 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-08-24 19:19:01 +00:00
mame 100bf27574 compile.c: remove tracecoverage instruction for line coverage
Line coverage was based on special instruction "tracecoverage".
Now, instead, it uses the mechanism of trace hook [Feature #14104].

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64509 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-08-22 11:09:47 +00:00
mame 33af0429ea thread.c (rb_reset_coverages): remove coverage counters from all ISeqs
When coverage measurement is enabled, the compiler makes each iseq have
a reference to the counter array of coverage.
Even after coverage measurement is disabled, the reference is kept.
And, if coverage measurement is restarted, a coverage hook will increase
the counter.  This is completely meaningless; it brings just overhead.

To remove this meaninglessness, this change removes all the reference
when coverage measuement is stopped.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64504 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-08-22 05:24:50 +00:00
normal 79bdb6f9c8 thread*.c: replace GetMutexPtr with mutex_ptr
Following ko1's lead in r59192, this gets rid of non-obvious
assignments which happen inside macros.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64490 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-08-21 01:01:42 +00:00
normal 39c64117f6 thread*.c: avoid unnecessary initialization for list_for_each_safe
According to r52446, it is only necessary for the current item (@i),
not the `@nxt` parameter for list_for_each_safe.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64486 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-08-20 21:34:44 +00:00
normal 17e4aff277 thread_pthread.c: reinitialize ubf_list at fork
It's possible for the ubf_list_head to be populated with dead
threads at fork or the ubf_list_lock to be held, so reinitialize
both at startup.

And while we're at it, use a static initializer at startup
to save a library call and kill some ifdef.

[ruby-core:88578] [Bug #15013]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64485 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-08-20 21:34:39 +00:00
normal 6056dae7f4 thread.c (rb_thread_fd_select): fix off-by-one with sigwait_fd
select(2) needs the nfds argument to be one higher than the
largest FD in the sets :x

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64475 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-08-19 20:40:42 +00:00
normal 77038f9f82 Revert "thread_sync.c (rb_mutex_sleep): skip interrupt check before sleep"
This reverts commit 2e420b8b99 (r64464)

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64466 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-08-19 00:01:03 +00:00
normal 2e420b8b99 thread_sync.c (rb_mutex_sleep): skip interrupt check before sleep
We do not want to risk switching threads before going to sleep
because it can cause unexpected wakeups and put us in an
unexpected state when used with ConditionVariable.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64464 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-08-18 20:04:07 +00:00
normal bfdf6cde9f Revert "thread.c (sleep_*): check interrupt before changing th->status"
This reverts commit 9e59487a38 (r64449)

More (but different) CI failures I can't reproduce locally...
http://ci.rvm.jp/results/trunk-test@ruby-sky3/1235951

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64451 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-08-18 19:03:44 +00:00
normal 9e59487a38 thread.c (sleep_*): check interrupt before changing th->status
Having threads switch before we sleep can cause applications
to misread the state of the thread.  Now, we are consistent
with blocking_region_begin behavior and change th->status
AFTER checking interrupts.

Maybe this can fix [Bug #15002]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64449 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-08-18 18:29:28 +00:00
normal fe2f89af9a thread.c (sleep_*): reduce the effect of spurious interrupts
Spurious interrupts from SIGCHLD cause Mutex#sleep (via
ConditionVariable#wait) to return early and breaks some use
cases.  Since these are outside the programs's control with
MJIT, we will only consider pending interrupts (e.g. those
from Thread#run) and signals which cause a Ruby-level Signal.trap
handler to fire as "spurious" wakeups.

[ruby-core:88537] [Feature #15002]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64444 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-08-18 09:07:36 +00:00
normal e3f6429eff thread.c (consume_communication_pipe): disarm UBF_TIMER before consume
Same reasoning as the disarm in rb_sigwait_fd_get, the current
thread is already processing signals, so we do not need
UBF_TIMER to continually kick the process, anymore.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64390 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-08-16 09:16:11 +00:00
normal 727a2d55a3 vm_core.h (rb_thread_t): pack small fields together
On a 64-bit system, this reduces rb_thread_t from 536 to 520 bytes.
Depending on the allocation, this can reduce cacheline access
for checking the abort_on_exception, report_on_exception and
pending_interrupt_queue_checked flags.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64376 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-08-15 05:54:41 +00:00
normal 48b6bd74e2 thread_pthread.c: eliminate timer thread by restructuring GVL
This reverts commit 194a6a2c68 (r64203).

Race conditions which caused the original reversion will be fixed
in the subsequent commit.

[ruby-core:88360] [Misc #14937]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64352 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-08-13 21:34:20 +00:00
normal 194a6a2c68 thread_pthread.c: restore timer-thread for now :<
[ruby-core:88306]

Revert "process.c: ensure th->interrupt lock is held when migrating"

This reverts commit 5ca416bdf6 (r64201)

Revert "process.c (rb_waitpid): reduce sigwait_fd bouncing"

This reverts commit 217bdd776f (r64200).

Revert "test/ruby/test_thread.rb (test_thread_timer_and_interrupt): add timeouts"

This reverts commit 9f395f1120 (r64199).

Revert "thread_pthread.c (native_sleep): reduce ppoll sleeps"

This reverts commit b3aa256c4d (r64193).

Revert "thread.c (consume_communication_pipe): do not retry after short read"

This reverts commit 291a82f748 (r64185).

Revert "test/ruby/test_io.rb (test_race_gets_and_close): timeout each thread"

This reverts commit 3dbd8d1f66 (r64184).

Revert "thread_pthread.c (gvl_acquire_common): persist timeout across calls"

This reverts commit 8c2ae6e3ed (r64165).

Revert "test/ruby/test_io.rb (test_race_gets_and_close): use SIGABRT on timeout"

This reverts commit 931cda4db8 (r64135).

Revert "thread_pthread.c (gvl_yield): do ubf wakeups when uncontended"

This reverts commit 508f00314f (r64133).

Revert "thread_pthread.h (native_thread_data): split condvars on some platforms"

This reverts commit a038bf238b (r64124).

Revert "process.c (waitpid_nogvl): prevent conflicting use of sleep_cond"

This reverts commit 7018acc946 (r64117).

Revert "thread_pthread.c (rb_sigwait_sleep): th may be 0 from MJIT"

This reverts commit 56491afc79 (r64116).

Revert "thread*.c: waiting on sigwait_fd performs periodic ubf wakeups"

This reverts commit ab47a57a46 (r64115).

Revert "thread_pthread.c (gvl_destroy): make no-op on GVL bits"

This reverts commit 95cae74817 (r64114).

Revert "thread_pthread.c (rb_sigwait_sleep): fix uninitialized poll set in UBF case"

This reverts commit 4514362948 (r64113).

Revert "thread_pthread.c (rb_sigwait_sleep): re-fix [Bug #5343] harder"

This reverts commit 26b8a70bb3 (r64111).

Revert "thread.c: move ppoll wrapper into thread_pthread.c"

This reverts commit 3dc7727d22 (r64110).

Revert "thread.c: move ppoll wrapper before thread_pthread.c"

This reverts commit 2fa1e2e3c3 (r64109).

Revert "thread_pthread.c (ubf_select): refix [Bug #5343]"

This reverts commit 4c1ab82f06 (r64108).

Revert "thread_win32.c: suppress warnings by -Wsuggest-attribute"

This reverts commit 6a9b63e390 (r64159).

Revert "thread_pthread: remove timer-thread by restructuring GVL"

This reverts commit 708bfd2115 (r64107).

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64203 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-08-06 05:22:00 +00:00
normal 291a82f748 thread.c (consume_communication_pipe): do not retry after short read
Saves a syscall and slightly improves vm_thread_condvar1
benchmark slightly (more improvements on the way):

                         r64170        this patch
vm_thread_condvar1        0.917        1.065

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64185 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-08-04 20:38:48 +00:00
normal ab47a57a46 thread*.c: waiting on sigwait_fd performs periodic ubf wakeups
We need to be able to perform periodic ubf_list wakeups when a
thread is sleeping and waiting on signals.

[ruby-core:88088] [Misc #14937] [Bug #5343]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64115 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-07-30 05:51:06 +00:00
normal 26b8a70bb3 thread_pthread.c (rb_sigwait_sleep): re-fix [Bug #5343] harder
We can't always designate a timer thread, so any sleepers must
also perform ubf wakeups.  Note: a similar change needs to be
made for rb_thread_fd_select and rb_wait_for_single_fd.

[ruby-core:88088] [Misc #14937] [Bug #5343]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64111 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-07-30 02:28:00 +00:00
normal 3dc7727d22 thread.c: move ppoll wrapper into thread_pthread.c
thread_pthread.c relies on ppoll for rb_sigwait_sleep, so ensure
the compatibility wrapper is available for it.

[Bug #14950]

Reported-by: SHIBATA Hiroshi <hsbt@ruby-lang.org>
Reported-by: Greg L <Greg.mpls@gmail.com>

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64110 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-07-30 00:31:08 +00:00
normal 2fa1e2e3c3 thread.c: move ppoll wrapper before thread_pthread.c
thread_pthread.c relies on ppoll for rb_sigwait_sleep, so ensure
the compatibility wrapper is available for it.

Reported-by: SHIBATA Hiroshi <hsbt@ruby-lang.org>

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64109 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-07-29 22:54:44 +00:00
normal 708bfd2115 thread_pthread: remove timer-thread by restructuring GVL
To reduce resource use and reduce CI failure; remove
timer-thread.  Single-threaded Ruby processes (including forked
children) will never see extra thread overhead.  This prevents
glibc and jemalloc from going into multi-threaded mode and
initializing locks or causing fragmentation via arena explosion.

The GVL is implements its own wait-queue as a ccan/list to
permit controlling wakeup order.  Timeslice under contention is
handled by a designated timer thread (similar to choosing a
"patrol_thread" for current deadlock checking).

There is only one self-pipe, now, as wakeups for timeslice are
done independently using condition variables.  This reduces FD
pressure slightly.

Signal handling is handled directly by a Ruby Thread (instead
of timer-thread) by exposing signal self-pipe to callers of
rb_thread_fd_select, native_sleep, rb_wait_for_single_fd, etc...
Acquiring, using, and releasing the self-pipe is exposed via 4
new internal functions:

1) rb_sigwait_fd_get - exclusively acquire timer_thread_pipe.normal[0]

2) rb_sigwait_fd_sleep - sleep and wait for signal (and no other FDs)

3) rb_sigwait_fd_put - release acquired result from rb_sigwait_fd_get

4) rb_sigwait_fd_migrate - migrate signal handling to another thread
                           after calling rb_sigwait_fd_put.

rb_sigwait_fd_migrate is necessary for waitpid callers because
only one thread can wait on self-pipe at a time, otherwise a
deadlock will occur if threads fight over the self-pipe.

TRAP_INTERRUPT_MASK is now set for the main thread directly in
signal handler via rb_thread_wakeup_timer_thread.

Originally, I wanted to use POSIX timers
(timer_create/timer_settime) for this.  Unfortunately, this
proved unfeasible as Mutex#sleep resumes on spurious wakeups and
test/thread/test_cv.rb::test_condvar_timed_wait failed.  Using
pthread_sigmask to mask out SIGVTALRM fixed that test,  but
test/fiddle/test_function.rb::test_nogvl_poll proved there'd be
some unavoidable (and frequent) incompatibilities from that
approach.

Finally, this allows us to drop thread_destruct_lock and
interrupt current ec directly.

We don't need to rely on vm->thread_destruct_lock or a coherent
vm->running_thread on any platform.  Separate timer-thread for
time slice and signal handling is relegated to thread_win32.c,
now.

[ruby-core:88088] [Misc #14937]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64107 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-07-29 20:47:33 +00:00
normal 856bd77aea thread.c (blocking_region_end): clear ubf before unregister_ubf_list
If we keep ubf set after unregistering, there is a window for
other threads (including timer thread) to put this thread back
on the ubf_list right away.  Entering ubf_list unexpectedly
after GVL acquisition may cause spurious wakeup and trigger
unexpected behavior.

Finally, clear ubf before acquiring GVL, to since ubf is useless
during GVL acquisition anyways and we don't want to waste cycles
in other threads calling ubf for useless work.

[ruby-core:88141] [Bug #14945]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64083 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-07-28 03:10:10 +00:00
normal 8a3ed368af Revert "thread.c (rb_thread_fd_select): remove unnecessary rb_fd_resize calls"
This reverts commit r64017
(git commit 2ff8562169).

Nevermind, haven't had enough coffee, yet :x

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64018 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-07-23 01:07:41 +00:00
normal 2ff8562169 thread.c (rb_thread_fd_select): remove unnecessary rb_fd_resize calls
There's no need to resize each rb_fdset_t to match the size of
the biggest one.  This can allow some small memory savings if
watching several sets of FDs simultaneously.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64017 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-07-23 00:51:51 +00:00
normal 8d0f5f1bfa thread.c (do_select): fix leak on exception
When do_select is interrupted and raise happens from
RUBY_VM_CHECK_INTS_BLOCKING, the original FD sets we copied
do not get freed, leading to a memory leak.  Wrap up all the
FD sets into a Ruby object to ensure the GC can release an
allocations made for rb_fdset_t.

This leak existed since Ruby 2.0.0 (r36430)

[Bug #14929]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64007 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-07-21 03:26:38 +00:00
normal c93adfc170 mjit: get rid of memory leak in pause+resume loop
pthread_atfork is not idempotent and repeatedly calling it
causes it to register the same hook repeatedly; leading to
unbound memory growth.

Ruby already has a (confusing-named) internal API for to call
in the forked child process: rb_thread_atfork
Call the MJIT child_after_fork hook inside that to prevent
unbound growth with the following loop:

    loop do
      RubyVM::MJIT.pause
      RubyVM::MJIT.resume
    end

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63884 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-07-08 07:27:24 +00:00
normal fcf8c622fd thread.c (ruby_ppoll): improve timespec to msec conversion
Round up non-zero <1ms timeouts to 1ms and use INT_MAX instead
of infinite (-1) for extremely large timeouts.  All of our
ppoll/select callers are able to handle spurious wakeups,
anyways.

This avoids excessive CPU usage and busy waits with short
timeouts to rb_wait_for_single_fd.

CPU usage with the following script is significantly reduced
for systems with "#undef HAVE_PPOLL":

    require 'io/wait'
    r, w = IO.pipe
    Thread.new { loop { r.wait_readable(0.000001) } }.join(5)
    exit!(0)

Low-resolution in poll(2) still sucks, though...

Note: I don't see the value in making a similar change to
time_timeval of a <1us sleep is attempted because of GVL
release and syscall latency.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63867 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-07-06 02:15:05 +00:00
normal 24aa7a13ca thread.c: our fake ppoll implementation is static
Rename it to "ruby_ppoll" so it looks more obvious in debuggers.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63866 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-07-06 02:15:00 +00:00
normal 44fc3d08eb unrevert r63852 but keep SIGCHLD path disabled for win32
Reading win32/win32.c waitpid implementation, maybe waitpid(-1, ...)
on that platform will never conflict with mjit use of waitpid.

In any case, I've added WAITPID_USE_SIGCHLD macro to vm_core.h
so it can be easy for Linux/BSD users to test (hopefully!)
win32-compatible code.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63855 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-07-05 03:02:33 +00:00
naruse df4a126d65 Revert r63758 and related commits
The change is unstable on Windows. Please re-commit it when it correctly
supports Windows.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63852 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-07-04 15:08:56 +00:00
normal 054a412d54 hijack SIGCHLD handler for internal use
Use a global SIGCHLD handler to guard all callers of rb_waitpid.
To work safely with multi-threaded programs, we introduce a
VM-wide waitpid_lock to be acquired BEFORE fork/vfork spawns the
process.  This is to be combined with the new ruby_waitpid_locked
function used by mjit.c in a non-Ruby thread.

Ruby-level SIGCHLD handlers registered with Signal.trap(:CHLD)
continues to work as before and there should be no regressions
in any existing use cases.

Splitting the wait queues for PID > 0 and groups (PID <= 0)
ensures we favor PID > 0 callers.

The disabling of SIGCHLD in rb_f_system is longer necessary,
as we use deferred signal handling and no longer make ANY
blocking waitpid syscalls in other threads which could "beat"
the waitpid call made by rb_f_system.

We prevent SIGCHLD from firing in normal Ruby Threads and only
enable it in the timer-thread, to prevent spurious wakeups
from in test/-ext-/gvl/test_last_thread.rb with MJIT enabled.

I've tried to guard as much of the code for RUBY_SIGCHLD==0
using C "if" statements rather than CPP "#if" so to reduce
the likelyhood of portability problems as the compiler will
see more code.

We also work to suppress false-positives from
Process.wait(-1, Process::WNOHANG) to quiets warnings from
spec/ruby/core/process/wait2_spec.rb with MJIT enabled.

Lastly, we must implement rb_grantpt for ext/pty.  We need a
MJIT-compatible way of supporting grantpt(3) which may spawn
the `pt_chown' binary and call waitpid(2) on it.

[ruby-core:87605] [Ruby trunk Bug#14867]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63758 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-06-27 03:14:30 +00:00
normal 3839734b94 thread.c (sleep_timespec): avoid needless update w/o spuriuos check
No point in wasting cycles updating the timespec when not
checking on spurious wakeups.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63719 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-06-22 06:17:15 +00:00
normal 986f11e72f thread.c (timeout_prepare): common function
I can't seem to reproduce the maybe-uninitialized warning on
gcc 7 or 8 on Debian sid (7.3.0-16 / 8-20180425-1 r259628),
so the guard from r62305 is dropped.

* thread.c (timeout_prepare): hoist out from do_select
  (do_select): ditto
  (rb_wait_for_single_fd): use timeout_prepare

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63672 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-06-16 08:27:56 +00:00
normal 5ff2a1968d thread.c: use flags for sleep_* functions
Same thing as https://bugs.ruby-lang.org/issues/14798
My easily-confused mind gets function call ordering confused
easily:

	sleep_forever(..., TRUE, FALSE);
	sleep_forever(..., FALSE, TRUE);

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63647 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-06-13 10:00:46 +00:00
normal 74724107e9 thread.c (rb_wait_for_single_fd): do not leak EINTR on timeout
We must not leak EINTR to users in case a signal hits a
ppoll/select caller right when (or just before) the timeout
expires.  In other words, the timeout should take precedence
over the -1 result from ppoll or select.

We also try one more time in case of EINTR with a zero timeout,
since technically the syscall finished before timing out if
it returns EINTR.

Regression appeared in r62457
("thread.c (update_timespec): use timespec_update_expire",
 commit e6bf0128ad)
and is not in any stable release of Ruby.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63462 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-05-18 08:01:07 +00:00
normal 0f0311df0a thread: reduce GET_THREAD calls
This allows native_sleep to use less stack (80 -> 64 bytes on
x86-64) for GVL_UNLOCK_BEGIN/END.  For future APIs, we will pass
`ec` or `th` around anyways, so the BLOCKING_REGION change
should be beneficial in the future.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63448 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-05-16 21:54:42 +00:00
normal 4e09f414fc thread.c: enable ppoll for FreeBSD 11.0 and later
FreeBSD 11.0+ supports ppoll, so we may use it after accounting
for portability differences in how it treats POLLOUT vs POLLHUP
events as mutually exclusive (as documented in the FreeBSD
poll(2) manpage).

For waiting on high-numbered single FDs, this should put
FreeBSD on equal footing with Linux and should allow cheaper
FD readiness checking with sleepy GC in the future.

* thread.c (USE_POLL, POLLERR_SET): define for FreeBSD 11.0+
  (rb_wait_for_single_fd): return all requested events on POLLERR_SET
  io.c (USE_POLL): define for FreeBSD 11.0+

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63427 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-05-15 03:49:21 +00:00
normal 45255c4233 use list_head_init instead of open-coding it
While we cannot use LIST_HEAD since r63312, we can at
least use list_head_init to make our code more readable.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63314 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-05-01 08:35:19 +00:00
shyouhei 496ddbc275 LIST_HEAD as a local variable is a C99ism.
Address of a variable whose storage duration is `auto` is _not_ a
compile time constant, according to ISO 9899 section 6.4.
LIST_HEAD takes such thing.  You can't use it to declare local
variables.

Interestingly, address of a static variable _is_ a compile time
constant.  So a declaration like `static LIST_HEAD..` is
completely legal even in C90.

In C99 and newer, this is not a constraint violation.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63312 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-05-01 04:41:10 +00:00
normal ce70bff315 thread.c (vm_living_thread_num): constify vm arg
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63307 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-04-30 23:12:03 +00:00
normal 645f7fbd4e io.c: do not use rb_notify_fd_close close on recycled FD
It is unsafe to release GVL and call rb_notify_fd_close after
close(2) on any given FD.  FDs (file descriptor) may be recycled
in other threads immediately after close() to point to a different
file description.  Note the distinction between "file description"
and "file descriptor".

th-1                           | th-2
-------------------------------+---------------------------------------
io_close_fptr                  |
  rb_notify_fd_close(fd)       |
  fptr_finalize_flush          |
    close(fd)                  |
  rb_thread_schedule           |
                               | fd reused (via pipe/open/socket/etc)
  rb_notify_fd_close(fd)       |
                               | sees "stream closed" exception
			       |   for DIFFERENT file description

* thread.c (rb_thread_io_blocking_region): adjust comment for list_del
* thread.c (rb_notify_fd_close): give busy list to caller
* thread.c (rb_thread_fd_close): loop on busy list
* io.c (io_close_fptr): do not call rb_thread_fd_close on invalid FD
* io.c (io_reopen): use rb_thread_fd_close

Fixes: r57422 ("io.c: close before wait")

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63216 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-04-21 03:12:36 +00:00
normal af72dcd91b thread_sync: redo r62934 to use fork_gen
Instead of maintaining linked-lists to store all
rb_queue/rb_szqueue/rb_condvar structs; store only a fork_gen
serial number to simplify management of these items.

This reduces initialization costs and avoids the up-front cost
of resetting all Queue/SizedQueue/ConditionVariable objects at
fork while saving 8 bytes per-structure on 64-bit.  There are no
savings on 32-bit.

* thread.c (rb_thread_atfork_internal): remove rb_thread_sync_reset_all call
* thread_sync.c (rb_thread_sync_reset_all): remove
* thread_sync.c (queue_live): remove
* thread_sync.c (queue_free): remove
* thread_sync.c (struct rb_queue): s/live/fork_gen/
* thread_sync.c (queue_data_type): use default free
* thread_sync.c (queue_alloc): remove list_add
* thread_sync.c (queue_fork_check): new function
* thread_sync.c (queue_ptr): call queue_fork_check
* thread_sync.c (szqueue_free): remove
* thread_sync.c (szqueue_data_type): use default free
* thread_sync.c (szqueue_alloc): remove list_add
* thread_sync.c (szqueue_ptr):  check fork_gen via queue_fork_check
* thread_sync.c (struct rb_condvar): s/live/fork_gen/
* thread_sync.c (condvar_free): remove
* thread_sync.c (cv_data_type): use default free
* thread_sync.c (condvar_ptr): check fork_gen
* thread_sync.c (condvar_alloc): remove list_add
  [ruby-core:86316] [Bug #14634]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63215 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-04-20 22:53:37 +00:00
normal 475b4aa40b simplify altstack and enable reuse with thread cache
Instead of allocating and registering the altstack in different
places, do it together to reduce code and improve readability.
When thread cache is enabled, storing altstack in rb_thread_t
is wasteful and we may reuse altstack in the same pthread.

This also lets us clearly allow use of xmalloc to allow GC to
recover from ENOMEM.

[ruby-core:85621] [Feature #14487]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63213 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-04-20 21:38:27 +00:00
normal b456eab2ea variable.c: fix thread + fork errors in autoload
This is fairly non-intrusive bugfix to prevent children
from trying to reach into thread stacks of the parent.
I will probably reuse this idea and redo r62934, too
(same bug).

* vm_core.h (typedef struct rb_vm_struct): add fork_gen counter
* thread.c (rb_thread_atfork_internal): increment fork_gen
* variable.c (struct autoload_data_i): store fork_gen
* variable.c (check_autoload_data): remove (replaced with get_...)
* variable.c (get_autoload_data): check fork_gen when retrieving
* variable.c (check_autoload_required): use get_autoload_data
* variable.c (rb_autoloading_value): ditto
* variable.c (rb_autoload_p): ditto
* variable.c (current_autoload_data): ditto
* variable.c (autoload_reset): reset fork_gen, adjust indent
* variable.c (rb_autoload_load): set fork_gen when setting state
* test/ruby/test_autoload.rb (test_autoload_fork): new test
  [ruby-core:86410] [Bug #14634]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63210 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-04-20 03:22:26 +00:00
ko1 51d227e3a5 refactoring r63073.
* cont.c (root_fiber_alloc): call `ConvertThreadToFiber()` here.

  `rb_fiber_t` for root_fiber is allocated before running Threads.
  Fiber objects wrapping this rb_fiber_t for root_fiber are created
  when root Fiber object is required explicitly (for example, Fiber
  switching and so on). We can put calling `ConvertThreadToFiber()`.
  In other words, we can pending `ConvertThreadToFiber()`
  until Fiber objects are created.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63090 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-04-04 08:19:28 +00:00
ko1 db3cc675f3 Fix Fiber with Thread issue on Windows [Bug #14642]
* cont.c (rb_threadptr_root_fiber_setup): divide into two functions:
  * rb_threadptr_root_fiber_setup_by_parent(): called by the parent thread.
  * rb_threadptr_root_fiber_setup_by_child(): called by the created thread.

  `rb_threadptr_root_fiber_setup()` is called by the parent thread and
  set fib->fib_handle by ConvertThreadToFiber() on the parent thread on
  Windows enveironment.
  This means that root_fib->fib_handle of child thread is initialized
  with parent thread's Fiber handle. Furthermore, second call of
  `ConvertThreadToFiber()` for the same thread fails.

  This patch solves this weird situateion. However, maybe we can make more
  clean code.

* thread.c (thread_start_func_2): call
  `rb_threadptr_root_fiber_setup_by_child()` at thread initialize routine.

* vm.c (th_init): call `rb_threadptr_root_fiber_setup_by_parent()`.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63073 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-04-03 10:21:47 +00:00
normal a2d63ea2fb thread_sync.c: avoid reaching across stacks of dead threads
rb_ensure is insufficient cleanup for fork and we must
reinitialize all waitqueues in the child process.

Unfortunately this increases the footprint of ConditionVariable,
Queue and SizedQueue by 8 bytes on 32-bit (16 bytes on 64-bit).

[ruby-core:86316] [Bug #14634]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62934 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-03-27 09:28:37 +00:00
nobu ec826b3f93 thread.c: deadlock in backtrace
* thread.c (unblock_function_set): check interrupts just once
  during raising exceptions, as they are deferred since r16651.
  [ruby-core:85939] [Bug #14577]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62673 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-03-06 05:15:57 +00:00
normal 1b455428d3 thread.c: reset waitq of keeping mutexes in child
We must not maintain references to threads in the parent process
in any mutexes held by the child process.

* thread_sync.c (rb_mutex_cleanup_keeping_mutexes): new function
* thread.c (rb_thread_atfork): cleanup keeping mutexes
  [ruby-core:85940] [Bug #14578]

Fixes: r58604 (commit 3586c9e087)
       ("reduce rb_mutex_t size from 160 to 80 bytes on 64-bit")

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62668 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-03-05 22:58:13 +00:00
nobu 96db72ce38 [DOC] missing docs at toplevel
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62544 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-02-23 02:18:52 +00:00
normal eac50243c9 thread.c: remove redundant USE_SIGALTSTACK #define
thread.c already includes vm_core.h where USE_SIGALTSTACK is
defined, #include it explicitly (eval_intern.h already includes
it)

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62473 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-02-18 21:03:13 +00:00
normal ecd2c08a4c thread.c (thread_join_m): handle negative timeouts correctly
Users may subtract and round into negative values when using
Thread#join, so clamp the timeout to zero to avoid infinite/long
timeouts.

Note: other methods such as Kernel#sleep and IO.select will
raise on negative values, but Thread#join is an outlier *shrug*

This restores Ruby 2.5 (and earlier) behavior.

Fixes: r62182 (commit c915390b95)
       ("thread.c: avoid FP for Thread#join")

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62462 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-02-18 03:00:33 +00:00
normal fbad2c5592 thread.c (double2timespec): adjust to use NULL for infinity
Using:
  strace ruby -e 'Thread.new { sleep }.join(Float::INFINITY)'

Will show a difference in futex() syscall args (not that I'd
ever advocate Float::INFINITY as a Thread#join arg :P)

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62461 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-02-18 03:00:28 +00:00
normal f65ee1e8f1 thread.c: stop updating timespec when timeout is unspecified
No need to waste cycles updating timespecs if there's no expiry.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62458 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-02-18 02:12:18 +00:00
normal e6bf0128ad thread.c (update_timespec): use timespec_update_expire
Rename "end" as a appropriate for readability.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62457 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-02-18 02:12:13 +00:00
normal 014e9a4a18 thread.c: introduce timespec_cmp for timespec comparisons
This hopefully improves readability when comparing timespecs.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62456 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-02-18 00:38:45 +00:00
normal b160a1139e thread.c (timespec_update_expire): improve naming
Naming the constant timespec as "end" should make it more
apparent is is an absolute time.  Update callers, too.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62455 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-02-18 00:38:40 +00:00
nobu c4d09fb299 thread.c: suppress a warning
* thread.c (do_select): initialize timespec variable to suppress a
  false positive maybe-uninitialized warning by gcc 7 and 8.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62305 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-02-08 07:49:42 +00:00
nobu 76c5787c38 thread.c: int32_t instead of suseconds_t
* thread.c (timeval_for): cast to int32_t instead of suseconds_t,
  which is not defined non-POSIX platforms.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62276 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-02-07 06:49:32 +00:00
nobu b0cf1c234c fix up r62272
* thread.c (timeval_for): tv_usec is suseconds_t which may be
  smaller than long.

* thread_pthread.c (native_cond_timeout): ret is now used in
  CLOCK_MONOTONIC case only.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62275 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-02-07 06:14:56 +00:00
normal 0abd9b7f25 thread.c: favor timespec internally
This results in fewer conversion on common modern systems with
support for clock_gettime, pthread_cond_timedwait and ppoll.
gettimeofday is declared obsolete by POSIX.1-2008, so it is yet
another reason to move away from it.  This also appears to result
in the reduction of compatibility code required for dealing
with inconsistent implementations of "struct timeval".tv_sec

In the future, this will also result in fewer conversions for
kqueue and pselect if we elect to use them.

[ruby-core:85416] [Feature #14452]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62272 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-02-07 01:57:14 +00:00
k0kubun ed935aa5be mjit_compile.c: merge initial JIT compiler
which has been developed by Takashi Kokubun <takashikkbn@gmail> as
YARV-MJIT. Many of its bugs are fixed by wanabe <s.wanabe@gmail.com>.

This JIT compiler is designed to be a safe migration path to introduce
JIT compiler to MRI. So this commit does not include any bytecode
changes or dynamic instruction modifications, which are done in original
MJIT.

This commit even strips off some aggressive optimizations from
YARV-MJIT, and thus it's slower than YARV-MJIT too. But it's still
fairly faster than Ruby 2.5 in some benchmarks (attached below).

Note that this JIT compiler passes `make test`, `make test-all`, `make
test-spec` without JIT, and even with JIT. Not only it's perfectly safe
with JIT disabled because it does not replace VM instructions unlike
MJIT, but also with JIT enabled it stably runs Ruby applications
including Rails applications.

I'm expecting this version as just "initial" JIT compiler. I have many
optimization ideas which are skipped for initial merging, and you may
easily replace this JIT compiler with a faster one by just replacing
mjit_compile.c. `mjit_compile` interface is designed for the purpose.

common.mk: update dependencies for mjit_compile.c.

internal.h: declare `rb_vm_insn_addr2insn` for MJIT.

vm.c: exclude some definitions if `-DMJIT_HEADER` is provided to
compiler. This avoids to include some functions which take a long time
to compile, e.g. vm_exec_core. Some of the purpose is achieved in
transform_mjit_header.rb (see `IGNORED_FUNCTIONS`) but others are
manually resolved for now. Load mjit_helper.h for MJIT header.
mjit_helper.h: New. This is a file used only by JIT-ed code. I'll
refactor `mjit_call_cfunc` later.
vm_eval.c: add some #ifdef switches to skip compiling some functions
like Init_vm_eval.

win32/mkexports.rb: export thread/ec functions, which are used by MJIT.

include/ruby/defines.h: add MJIT_FUNC_EXPORTED macro alis to clarify
that a function is exported only for MJIT.

array.c: export a function used by MJIT.
bignum.c: ditto.
class.c: ditto.
compile.c: ditto.
error.c: ditto.
gc.c: ditto.
hash.c: ditto.
iseq.c: ditto.
numeric.c: ditto.
object.c: ditto.
proc.c: ditto.
re.c: ditto.
st.c: ditto.
string.c: ditto.
thread.c: ditto.
variable.c: ditto.
vm_backtrace.c: ditto.
vm_insnhelper.c: ditto.
vm_method.c: ditto.

I would like to improve maintainability of function exports, but I
believe this way is acceptable as initial merging if we clarify the
new exports are for MJIT (so that we can use them as TODO list to fix)
and add unit tests to detect unresolved symbols.
I'll add unit tests of JIT compilations in succeeding commits.

Author: Takashi Kokubun <takashikkbn@gmail.com>
Contributor: wanabe <s.wanabe@gmail.com>

Part of [Feature #14235]

---

* Known issues
  * Code generated by gcc is faster than clang. The benchmark may be worse
    in macOS. Following benchmark result is provided by gcc w/ Linux.
  * Performance is decreased when Google Chrome is running
  * JIT can work on MinGW, but it doesn't improve performance at least
    in short running benchmark.
  * Currently it doesn't perform well with Rails. We'll try to fix this
    before release.

---

* Benchmark reslts

Benchmarked with:
Intel 4.0GHz i7-4790K with 16GB memory under x86-64 Ubuntu 8 Cores

- 2.0.0-p0: Ruby 2.0.0-p0
- r62186: Ruby trunk (early 2.6.0), before MJIT changes
- JIT off: On this commit, but without `--jit` option
- JIT on: On this commit, and with `--jit` option

** Optcarrot fps

Benchmark: https://github.com/mame/optcarrot

|         |2.0.0-p0 |r62186   |JIT off  |JIT on   |
|:--------|:--------|:--------|:--------|:--------|
|fps      |37.32    |51.46    |51.31    |58.88    |
|vs 2.0.0 |1.00x    |1.38x    |1.37x    |1.58x    |

** MJIT benchmarks

Benchmark: https://github.com/benchmark-driver/mjit-benchmarks
(Original: https://github.com/vnmakarov/ruby/tree/rtl_mjit_branch/MJIT-benchmarks)

|           |2.0.0-p0 |r62186   |JIT off  |JIT on   |
|:----------|:--------|:--------|:--------|:--------|
|aread      |1.00     |1.09     |1.07     |2.19     |
|aref       |1.00     |1.13     |1.11     |2.22     |
|aset       |1.00     |1.50     |1.45     |2.64     |
|awrite     |1.00     |1.17     |1.13     |2.20     |
|call       |1.00     |1.29     |1.26     |2.02     |
|const2     |1.00     |1.10     |1.10     |2.19     |
|const      |1.00     |1.11     |1.10     |2.19     |
|fannk      |1.00     |1.04     |1.02     |1.00     |
|fib        |1.00     |1.32     |1.31     |1.84     |
|ivread     |1.00     |1.13     |1.12     |2.43     |
|ivwrite    |1.00     |1.23     |1.21     |2.40     |
|mandelbrot |1.00     |1.13     |1.16     |1.28     |
|meteor     |1.00     |2.97     |2.92     |3.17     |
|nbody      |1.00     |1.17     |1.15     |1.49     |
|nest-ntimes|1.00     |1.22     |1.20     |1.39     |
|nest-while |1.00     |1.10     |1.10     |1.37     |
|norm       |1.00     |1.18     |1.16     |1.24     |
|nsvb       |1.00     |1.16     |1.16     |1.17     |
|red-black  |1.00     |1.02     |0.99     |1.12     |
|sieve      |1.00     |1.30     |1.28     |1.62     |
|trees      |1.00     |1.14     |1.13     |1.19     |
|while      |1.00     |1.12     |1.11     |2.41     |

** Discourse's script/bench.rb

Benchmark: https://github.com/discourse/discourse/blob/v1.8.7/script/bench.rb

NOTE: Rails performance was somehow a little degraded with JIT for now.
We should fix this.
(At least I know opt_aref is performing badly in JIT and I have an idea
 to fix it. Please wait for the fix.)

*** JIT off
Your Results: (note for timings- percentile is first, duration is second in millisecs)

categories_admin:
  50: 17
  75: 18
  90: 22
  99: 29
home_admin:
  50: 21
  75: 21
  90: 27
  99: 40
topic_admin:
  50: 17
  75: 18
  90: 22
  99: 32
categories:
  50: 35
  75: 41
  90: 43
  99: 77
home:
  50: 39
  75: 46
  90: 49
  99: 95
topic:
  50: 46
  75: 52
  90: 56
  99: 101

*** JIT on
Your Results: (note for timings- percentile is first, duration is second in millisecs)

categories_admin:
  50: 19
  75: 21
  90: 25
  99: 33
home_admin:
  50: 24
  75: 26
  90: 30
  99: 35
topic_admin:
  50: 19
  75: 20
  90: 25
  99: 30
categories:
  50: 40
  75: 44
  90: 48
  99: 76
home:
  50: 42
  75: 48
  90: 51
  99: 89
topic:
  50: 49
  75: 55
  90: 58
  99: 99

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62197 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-02-04 11:22:28 +00:00
k0kubun fd44a5777f mjit.c: merge MJIT infrastructure
that allows to JIT-compile Ruby methods by generating C code and
using C compiler.  See the first comment of mjit.c to know what this
file does.

mjit.c is authored by Vladimir Makarov <vmakarov@redhat.com>.
After he invented great method JIT infrastructure for MRI as MJIT,
Lars Kanis <lars@greiz-reinsdorf.de> sent the patch to support MinGW
in MJIT. In addition to merging it, I ported pthread to Windows native
threads. Now this MJIT infrastructure can be compiled on Visual Studio.

This commit simplifies mjit.c to decrease code at initial merge. For
example, this commit does not provide multiple JIT threads support.
We can resurrect them later if we really want them, but I wanted to minimize
diff to make it easier to review this patch.

`/tmp/_mjitXXX` file is renamed to `/tmp/_ruby_mjitXXX` because non-Ruby
developers may not know the name "mjit" and the file name should make
sure it's from Ruby and not from some harmful programs.  TODO: it may be
better to store this to some temporary directory which Ruby is already using
by Tempfile, if it's not bad for performance.

mjit.h: New. It has `mjit_exec` interface similar to `vm_exec`, which is
for triggering MJIT. This drops interface for AOT compared to the original
MJIT.

Makefile.in: define macros to let MJIT know the path of MJIT header.
Probably we can refactor this to reduce the number of macros (TODO).
win32/Makefile.sub: ditto.

common.mk: compile mjit.o and mjit_compile.o. Unlike original MJIT, this
commit separates MJIT infrastructure and JIT compiler code as independent
object files. As initial patch is NOT going to have ultra-fast JIT compiler,
it's likely to replace JIT compiler, e.g. original MJIT's compiler or some
future JIT impelementations which are not public now.

inits.c: define MJIT module. This is added because `MJIT.enabled?` was
necessary for testing.
test/lib/zombie_hunter.rb: skip if `MJIT.enabled?`. Obviously this
wouldn't work with current code when JIT is enabled.
test/ruby/test_io.rb: skip this too. This would make no sense with MJIT.

ruby.c: define MJIT CLI options. As major difference from original MJIT,
"-j:l"/"--jit:llvm" are renamed to "--jit-cc" because I want to support
not only gcc/clang but also cl.exe (Visual Studio) in the future. But it
takes only "--jit-cc=gcc", "--jit-cc=clang" for now. And only long "--jit"
options are allowed since some Ruby committers preferred it at Ruby
developers Meeting on January, and some of options are renamed.
This file also triggers to initialize MJIT thread and variables.
eval.c: finalize MJIT worker thread and variables.
test/ruby/test_rubyoptions.rb: fix number of CLI options for --jit.

thread_pthread.c: change for pthread abstraction in MJIT. Prefix rb_ for
functions which are used by other files.
thread_win32.c: ditto, for Windows.  Those pthread porting is one of major
works that YARV-MJIT created, which is my fork of MJIT, in Feature 14235.
thread.c: follow rb_ prefix changes

vm.c: trigger MJIT call on VM invocation. Also trigger `mjit_mark` to avoid
SEGV by race between JIT and GC of ISeq. The improvement was provided by
wanabe <s.wanabe@gmail.com>.
In JIT compiler I created and am going to add in my next commit, I found
that having `mjit_exec` after `vm_loop_start:` is harmful because the
JIT-ed function doesn't proceed other ISeqs on RESTORE_REGS of leave insn.
Executing non-FINISH frame is unexpected for my JIT compiler and
`exception_handler` triggers executions of such ISeqs. So `mjit_exec`
here should be executed only when it directly comes from `vm_exec` call.
`RubyVM::MJIT` module and `.enabled?` method is added so that we can skip
some tests which don't expect JIT threads or compiler file descriptors.

vm_insnhelper.h: trigger MJIT on method calls during VM execution.

vm_core.h: add fields required for mjit.c. `bp` must be `cfp[6]` because
rb_control_frame_struct is likely to be casted to another struct. The
last position is the safest place to add the new field.
vm_insnhelper.c: save initial value of cfp->ep as cfp->bp. This is an
optimization which are done in both MJIT and YARV-MJIT. So this change
is added in this commit. Calculating bp from ep is a little heavy work,
so bp is kind of cache for it.

iseq.c: notify ISeq GC to MJIT. We should know which iseq in MJIT queue
is GCed to avoid SEGV.  TODO: unload some GCed units in some safe way.

gc.c: add hooks so that MJIT can wait GC, and vice versa. Simultaneous
JIT and GC executions may cause SEGV and so we should synchronize them.

cont.c: save continuation information in MJIT worker. As MJIT shouldn't
unload JIT-ed code which is being used, MJIT wants to know full list of
saved execution contexts for continuation and detect ISeqs in use.

mjit_compile.c: added empty JIT compiler so that you can reuse this commit
to build your own JIT compiler. This commit tries to compile ISeqs but
all of them are considered as not supported in this commit. So you can't
use JIT compiler in this commit yet while we added --jit option now.

Patch author: Vladimir Makarov <vmakarov@redhat.com>.

Contributors:
Takashi Kokubun <takashikkbn@gmail.com>.
wanabe <s.wanabe@gmail.com>.
Lars Kanis <lars@greiz-reinsdorf.de>.

Part of Feature 12589 and 14235.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62189 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-02-04 06:58:09 +00:00
nobu 0718f53bca thread.c: timespec_for is used only if poll() is used
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62186 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-02-04 01:31:48 +00:00
normal 05c18139a1 thread.c (thread_join_m): avoid NUM2TIMET for Bignum
Bignums exceed the range of time_t (or long).

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62184 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-02-03 22:24:17 +00:00
normal c0e2f98f9b thread.c: avoid FP in C-API time calculations
FP arithmetic can lose precision in some cases leading to
premature wakeup and wasting CPU cycles.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62183 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-02-03 19:59:21 +00:00
normal c915390b95 thread.c: avoid FP for Thread#join
FP arithmetic can lose precision in some cases leading to
premature wakeup and wasting CPU cycles.

Convert to use timeval_* functions for now.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62182 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-02-03 19:59:16 +00:00
normal 712439b530 thread.c: extract timeval_sub from timeval_update_expire
It can be useful on its own.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62181 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-02-03 19:59:11 +00:00
normal 076d886398 thread.c (rb_thread_terminate_all): eliminate double2timeval call
No point for a fixed 1s value, and I plan on eliminating double
timeouts from internal API.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62179 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-02-03 19:59:05 +00:00
hsbt 5af43b1ec2 ia64: update ia64-specific code to use execution context
This change follows commit 837fd5e494
in '#ifdef __ia64' branches.

Noticed as a build failure by John Paul Adrian Glaubitz:

```
  cont.c:502:50: error: 'rb_thread_t {aka struct rb_thread_struct}'
    has no member named 'machine'
       size = cont->machine.register_stack_size =
           th->machine.register_stack_end - th->machine.register_stack_start;
               ^~
```

The change is trivial: update 'th->machine' usage to 'th->ec->machine'.
Signed-off-by: Sergei Trofimovich <slyfox@gentoo.org>

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62106 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-01-30 05:43:42 +00:00
normal 72b7bd0af5 vm_core: use "int" for living_thread_num
We treat this as "int" through the vm_living_thread_num API
anyways, and "pid_t" is still 32-bits with glibc on 64-bit
platforms.  I expect it'll be a long time before anybody needs
more than 2 billion native threads.  For now, let's save one
cacheline on x86-64 (as reported by pahole(1)):

	before:  size: 1288, cachelines: 21, members: 45
	 after:  size: 1280, cachelines: 20, members: 45

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62075 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-01-27 21:04:36 +00:00
shyouhei 3b787be52f rb_threadptr_to_kill marked as NORETURN
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61925 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-01-18 09:44:46 +00:00
hsbt 9869ec7a7d Fix typos.
* node.c: strucutre -> structure
 * random.c: acquried -> acquired
 * thread.c: accross -> across

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61889 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-01-17 13:15:13 +00:00
normal 99886a9121 thread.c (thread_cleanup_func): document small leak
It's minor, I haven't analyzed how fixable it is, but we should
at least note it, here.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61779 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-01-12 00:15:28 +00:00
normal 3e8b9f85ff thread.c: threadptr_pending_interrupt_active_p is static
It's not used elsewhere.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61749 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-01-09 22:59:58 +00:00
kazu 51edb30042 Include ruby/{io,encoding}.h before internal.h
because of r61712 and r61713

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61725 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-01-09 12:35:12 +00:00
mame 4928900814 Rename code_range to code_location
Because the name "code_range" is ambiguous with encoding's.
Abbreviations ("crange", and "cr") are also renamed to "loc".

The traditional "code_location" (a pair of lineno and column) is
renamed to "code_position".  Abbreviations are also renamed
(first_loc to beg_pos, and last_loc to end_pos).

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61721 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-01-09 08:45:35 +00:00
shyouhei b6a2d63eb3 explicit cast to void* required for %p
These functions take variadic arguments so no automatic type
promotion is expected.  You have to do it by hand.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61542 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-01-02 06:41:40 +00:00
ko1 c39bdb798d `$SAFE` as a process global state. [Feature #14250]
* vm_core.h (rb_vm_t): move `rb_execution_context_t::safe_level` to
  `rb_vm_t::safe_level_` because `$SAFE` is a process (VM) global state.

* vm_core.h (rb_proc_t): remove `rb_proc_t::safe_level` because `Proc`
  objects don't need to keep `$SAFE` at the creation.
  Also make `is_from_method` and `is_lambda` as 1 bit fields.

* cont.c (cont_restore_thread): no need to keep `$SAFE` for Continuation.

* eval.c (ruby_cleanup): use `rb_set_safe_level_force()` instead of access
  `vm->safe_level_` directly.

* eval_jump.c: End procs `END{}` doesn't keep `$SAFE`.

* proc.c (proc_dup): removed and introduce `rb_proc_dup` in vm.c.

* safe.c (rb_set_safe_level): don't check `$SAFE` 1 -> 0 changes.

* safe.c (safe_setter): use `rb_set_safe_level()`.

* thread.c (rb_thread_safe_level): `Thread#safe_level` returns `$SAFE`.
  It should be obsolete.

* transcode.c (load_transcoder_entry): `rb_safe_level()` only returns
  0 or 1 so that this check is not needed.

* vm.c (vm_proc_create_from_captured): don't need to keep `$SAFE` for Proc.

* vm.c (rb_proc_create): renamed to `proc_create`.

* vm.c (rb_proc_dup): moved from proc.c.

* vm.c (vm_invoke_proc): do not need to set and restore `$SAFE`
  for `Proc#call`.

* vm_eval.c (rb_eval_cmd): rename a local variable to represent clearer
  meaning.

* lib/drb/drb.rb: restore `$SAFE`.

* lib/erb.rb: restore `$SAFE`, too.

* test/lib/leakchecker.rb: check `$SAFE == 0` at the end of tests.

* test/rubygems/test_gem.rb: do not set `$SAFE = 1`.

* bootstraptest/test_proc.rb: catch up this change.

* spec/ruby/optional/capi/string_spec.rb: ditto.

* test/bigdecimal/test_bigdecimal.rb: ditto.

* test/fiddle/test_func.rb: ditto.

* test/fiddle/test_handle.rb: ditto.

* test/net/imap/test_imap_response_parser.rb: ditto.

* test/pathname/test_pathname.rb: ditto.

* test/readline/test_readline.rb: ditto.

* test/ruby/test_file.rb: ditto.

* test/ruby/test_optimization.rb: ditto.

* test/ruby/test_proc.rb: ditto.

* test/ruby/test_require.rb: ditto.

* test/ruby/test_thread.rb: ditto.

* test/rubygems/test_gem_specification.rb: ditto.

* test/test_tempfile.rb: ditto.

* test/test_tmpdir.rb: ditto.

* test/win32ole/test_win32ole.rb: ditto.

* test/win32ole/test_win32ole_event.rb: ditto.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61510 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-12-28 20:09:24 +00:00
kazu 220d2a7182 Fix KeyError#{key,receiver} of Thread#fetch
[ruby-core:84508] [Bug #14247]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61507 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-12-28 00:00:05 +00:00
mame c08e8886ba compile.c: add a RUBY_EVENT_COVERAGE_LINE event for line coverage
2.5's line coverage measurement was about two times slower than 2.4
because of two reasons; (1) vm_trace uses rb_iseq_event_flags (which
takes O(n) currently where n is the length of iseq) to get an event
type, and (2) RUBY_EVENT_LINE uses setjmp to call an event hook.

This change adds a special event for line coverage,
RUBY_EVENT_COVERAGE_LINE, and adds `tracecoverage` instructions where
the event occurs in iseq.
`tracecoverage` instruction calls an event hook without vm_trace.
And, RUBY_EVENT_COVERAGE_LINE is an internal event which does not
use setjmp.

This change also cancells lineno change due to the deletion of trace
instructions [Feature #14104].  So fixes [Bug #14191].

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61350 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-12-20 04:24:14 +00:00
nobu cb26d0d9bb thread.c: adjusted [ci skip]
* thread.c (timeval_add): adjusted indent and parenthesized in
  braces.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61303 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-12-17 09:40:19 +00:00
normal db9523ef47 thread.c: fix overly long Thread#join w/ timeout
* test/ruby/test_thread.rb (test_signal_at_join): test with timeout
* thread.c (sleep_wait_for_interrupt): remove
  (thread_join_sleep): use native_sleep directly to avoid extra
  missing thread status change
  [Bug #14181]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61302 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-12-17 08:26:17 +00:00
normal 78cc1491d5 thread.c: hoist out timeval arithmetic functions
timeval arithmetic may be reused in other places and this
makes sleep_timeval easier-to-read.

* thread.c (timeval_add): hoist out of sleep_timeval
  (timeval_update_expire): ditto
  (sleep_timeval): use new functions

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61301 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-12-17 08:26:12 +00:00