While profiling msgpack-ruby I noticed a very substantial amout of time
spent in `rb_enc_associate_index`, called by `rb_utf8_str_new`.
On that benchmark, `rb_utf8_str_new` is 33% of the total runtime,
in big part because it cause GC to trigger often, but even then
`5.3%` of the total runtime is spent in `rb_enc_associate_index`
called by `rb_utf8_str_new`.
After closer inspection, it appears that it's performing a lot of
safety check we can assert we don't need, and other extra useless
operations, because strings are first created and filled as ASCII-8BIT
and then later reassociated to the desired encoding.
By directly allocating the string with the right encoding, it allow
to skip a lot of duplicated and useless operations.
After this change, the time spent in `rb_utf8_str_new` is down
to `28.4%` of total runtime, and most of that is GC.
[Bug #20598]
Just like [Bug #20595], Encoding#name_list and Encoding#aliases can have
their strings corrupted when Encoding.default_internal is set to nil.
Co-authored-by: Matthew Valentine-House <matt@eightbitraptor.com>
[Bug #20595]
enc_set_default_encoding will free the C string if the encoding is nil,
but the C string can be used by the encoding name string. This will cause
the encoding name string to be corrupted.
Consider the following code:
Encoding.default_internal = Encoding::ASCII_8BIT
names = Encoding.default_internal.names
p names
Encoding.default_internal = nil
p names
It outputs:
["ASCII-8BIT", "BINARY", "internal"]
["ASCII-8BIT", "BINARY", "\x00\x00\x00\x00\x00\x00\x00\x00"]
Co-authored-by: Matthew Valentine-House <matt@eightbitraptor.com>
[Feature #18576]
Since outright renaming `ASCII-8BIT` is deemed to backward incompatible,
the next best thing would be to only change its `#inspect`, particularly
in exception messages.
This `st_table` is used to both mark and pin classes
defined from the C API. But `vm->mark_object_ary` already
does both much more efficiently.
Currently a Ruby process starts with 252 rooted classes,
which uses `7224B` in an `st_table` or `2016B` in an `RArray`.
So a baseline of 5kB saved, but since `mark_object_ary` is
preallocated with `1024` slots but only use `405` of them,
it's a net `7kB` save.
`vm->mark_object_ary` is also being refactored.
Prior to this changes, `mark_object_ary` was a regular `RArray`, but
since this allows for references to be moved, it was marked a second
time from `rb_vm_mark()` to pin these objects.
This has the detrimental effect of marking these references on every
minors even though it's a mostly append only list.
But using a custom TypedData we can save from having to mark
all the references on minor GC runs.
Addtionally, immediate values are now ignored and not appended
to `vm->mark_object_ary` as it's just wasted space.
There is a memory leak in Encoding.default_external= and
Encoding.default_internal= because the duplicated name is not freed
when overwriting.
10.times do
1_000_000.times do
Encoding.default_internal = nil
end
puts `ps -o rss= -p #{$$}`
end
Before:
25664
41504
57360
73232
89168
105056
120944
136816
152720
168576
After:
9648
9648
9648
9680
9680
9680
9680
9680
9680
9680
when the RUBY_FREE_ON_SHUTDOWN environment variable is set, manually free memory at shutdown.
Co-authored-by: Nobuyoshi Nakada <nobu@ruby-lang.org>
Co-authored-by: Peter Zhu <peter@peterzhu.ca>
Instead of relying on setting an unsetting ruby_verbose, which is
not thread-safe, restructure require_internal and load_lock to
accept a warn argument for whether to warn, and add
rb_require_internal_silent to require without warnings. Use
rb_require_internal_silent when loading encoding.
Note this does not modify ruby_debug and errinfo handling, those
remain thread-unsafe.
Also silent requires when loading transcoders.
rb_enc_from_index(index) doesn't need locking if index specify
ASCII/US_ASCCII/UTF-8.
rb_enc_from_index() is called frequently so it has impact.
user system total real
r_parallel/miniruby 174 0.000209 0.000000 5.559872 ( 1.811501)
r_parallel/master_mini 175 0.000238 0.000000 12.664707 ( 3.523641)
(repeat x1000 `s.split(/,/)` where s = '0,,' * 1000)
* Use UTF-8 as default for Encoding.default_external on Windows
* Document UTF-8 change on Windows to Encoding.default_external
fix https://bugs.ruby-lang.org/issues/16604
To make some kind of Ractor related extensions, some functions
should be exposed.
* include/ruby/thread_native.h
* rb_native_mutex_*
* rb_native_cond_*
* include/ruby/ractor.h
* RB_OBJ_SHAREABLE_P(obj)
* rb_ractor_shareable_p(obj)
* rb_ractor_std*()
* rb_cRactor
and rm ractor_pub.h
and rename srcdir/ractor.h to srcdir/ractor_core.h
(to avoid conflict with include/ruby/ractor.h)
This reverts commit de17e2dea1.
This patch can introduce race condition because of conflicting
read/write access for enc_table::default_list. Maybe we need to
freeze default_list at the end of Init_encdb() in enc/encdb.c.
To reduce the number of locking for encoding manipulation,
enc_table::list is splited to ::default_list and ::additional_list.
::default_list is pre-allocated and no need locking to access to
the ::default_list. If additional encoding space is needed, use
::additional_list and this list need to use locking.
However, most of case, ::default_list is enough.
enc_table which manages Encoding information. rb_encoding_list
also manages Encoding objects. Both are accessed/modified by ractors
simultaneously so that they should be synchronized.
For enc_table, this patch introduced GLOBAL_ENC_TABLE_ENTER/LEAVE/EVAL
to access this table with VM lock. To make shortcut, three new global
variables global_enc_ascii, global_enc_utf_8, global_enc_us_ascii are
also introduced.
For rb_encoding_list, we split it to rb_default_encoding_list (256 entries)
and rb_additional_encoding_list. rb_default_encoding_list is fixed sized Array
so we don't need to synchronized (and most of apps only needs it). To manage
257 or more encoding objects, they are stored into rb_additional_encoding_list.
To access rb_additional_encoding_list., VM lock is needed.
After the encoding index instance variable is removed when all
instance variables are removed in `obj_free`, then `rb_str_free`
causes uninitialized instance variable warning and nil-to-integer
conversion exception. Both cases result in object allocation
during GC, and crashes.
This `str2` variable can be non-string (regexp etc.) but the previous
code passed it directly to rb_enc_str_asciionly_p(), which expects its
argument be a string. Let's enforce that constraint.
Data written to files is not transcoded per default, but only
when default_internal is set.
The default for default_internal is nil and doesn't depend on the
source file encoding.
Saves comitters' daily life by avoid #include-ing everything from
internal.h to make each file do so instead. This would significantly
speed up incremental builds.
We take the following inclusion order in this changeset:
1. "ruby/config.h", where _GNU_SOURCE is defined (must be the very
first thing among everything).
2. RUBY_EXTCONF_H if any.
3. Standard C headers, sorted alphabetically.
4. Other system headers, maybe guarded by #ifdef
5. Everything else, sorted alphabetically.
Exceptions are those win32-related headers, which tend not be self-
containing (headers have inclusion order dependencies).
This removes the related tests, and puts the related specs behind
version guards. This affects all code in lib, including some
libraries that may want to support older versions of Ruby.
This removes the security features added by $SAFE = 1, and warns for access
or modification of $SAFE from Ruby-level, as well as warning when calling
all public C functions related to $SAFE.
This modifies some internal functions that took a safe level argument
to no longer take the argument.
rb_require_safe now warns, rb_require_string has been added as a
version that takes a VALUE and does not warn.
One public C function that still takes a safe level argument and that
this doesn't warn for is rb_eval_cmd. We may want to consider
adding an alternative method that does not take a safe level argument,
and warn for rb_eval_cmd.
This changeset basically replaces `ruby_xmalloc(x * y)` into
`ruby_xmalloc2(x, y)`. Some convenient functions are also
provided for instance `rb_xmalloc_mul_add(x, y, z)` which allocates
x * y + z byes.
"locale encoding" is misleading since it doesn't mean Encoding.find("locale")
but the encoding used to interpret the script file. It's therefore better to
call it "script encoding" as in the paragraphs above.
Closes: https://github.com/ruby/ruby/pull/1611
Jean Boussier
Nobuyoshi Nakada
Peter Zhu
Jean Boussier
Adam Hess
Benoit Daloze
Koichi Sasada
Takashi Kokubun
Burdette Lamar
卜部昌平
Jeremy Evans
S-H-GAMELINKS
Lars Kanis
Stefan Stüben
QuestionDriven
Lars Kanis
Lourens Naudé