The test fails when RGENGC_CHECK_MODE is turned on:
1) Failure:
TestSymbol#test_inspect_under_gc_compact_stress [test/ruby/test_symbol.rb:123]:
<":testing"> expected but was
<":\x00\x00\x00\x00\x00\x00\x00">.
The test fails when RGENGC_CHECK_MODE is turned on:
TestString#test_sub_gc_compact_stress = 9.42 s
1) Failure:
TestString#test_sub_gc_compact_stress [test/ruby/test_string.rb:2089]:
<"aaa [amp] yyy"> expected but was
<"aaa [] yyy">.
Embedded shared strings cannot be moved because strings point into the
slot of the shared string. There may be code using the RSTRING_PTR on
the stack, which would pin the string but not pin the shared string,
causing it to move.
`String#+@` is 2-3 times faster than `String#dup` because it can
directly go through `rb_str_dup` instead of using the generic
much slower `rb_obj_dup`.
This fact led to the existance of the ugly `Performance/UnfreezeString`
rubocop performance rule that encourage users to rewrite the much
more readable and convenient `"foo".dup` into the ugly `(+"foo")`.
Let's make that rubocop rule useless.
```
compare-ruby: ruby 3.3.0dev (2023-11-20T02:02:55Z master 701b0650de) [arm64-darwin22]
last_commit=[ruby/prism] feat: add encoding for IBM865 (https://github.com/ruby/prism/pull/1884)
built-ruby: ruby 3.3.0dev (2023-11-20T12:51:45Z faster-str-lit-dup 6b745bbc5d) [arm64-darwin22]
warming up..
| |compare-ruby|built-ruby|
|:------|-----------:|---------:|
|uplus | 16.312M| 16.332M|
| | -| 1.00x|
|dup | 5.912M| 16.329M|
| | -| 2.76x|
```
Some code out there blind calls `force_encoding` without checking
what the original encoding was, which clears the coderange uselessly.
If the String is big, it can be a rather costly mistake.
For instance the `rack-utf8_sanitizer` gem does this on request
bodies.
Previously we used the next character following the found prefix to
determine if the match ended on a broken character.
This had caused surprising behaviour when a valid character was followed
by a UTF-8 continuation byte.
This commit changes the behaviour to instead look for the end of the
last character in the prefix.
[Bug #19784]
Co-authored-by: ywenc <ywenc@github.com>
Co-authored-by: Nobuyoshi Nakada <nobu@ruby-lang.org>
Previously, the following crashed due to use-after-free
with AArch64 Alpine Linux 3.18.3 (aarch64-linux-musl):
```ruby
str = 'a' * (32*1024*1024)
p({z: str})
```
32 MiB is the default for `GC_MALLOC_LIMIT_MAX`, and the crash
could be dodged by setting `RUBY_GC_MALLOC_LIMIT_MAX` to large values.
Under a debugger, one can see the `str2` of rb_str_buf_append()
getting prematurely collected while str_buf_cat4() allocates capacity.
Add GC guards so the buffer of `str2` lives across the GC run
initiated in str_buf_cat4().
[Bug #19792]
Leave callers to convert byte index to char index, as well as
`rb_str_index`, so that `rb_str_rpartition` does not need to
re-convert char index to byte index.
When String#split is used with an empty string as the field seperator it
effectively splits the original string into chars, and there is a
pre-existing fast path for this using SPLIT_TYPE_CHARS.
However this path creates an empty array in the smallest size pool and
grows from there, despite already knowing the size of the desired array.
This commit pre-allocates the correct size array in this case in order
to allow the arrays to be embedded and avoid being allocated in the
transient heap
* Unify length field for embedded and heap strings
The length field is of the same type and position in RString for both
embedded and heap allocated strings, so we can unify it.
* Remove RSTRING_EMBED_LEN
Remove !USE_RVARGC code
[Feature #19579]
The Variable Width Allocation feature was turned on by default in Ruby
3.2. Since then, we haven't received bug reports or backports to the
non-Variable Width Allocation code paths, so we assume that nobody is
using it. We also don't plan on maintaining the non-Variable Width
Allocation code, so we are going to remove it.
bytesplice(index, length, str, str_index, str_length) -> string
bytesplice(range, str, str_range) -> string
In these forms, the content of +self+ is replaced by str.byteslice(str_index, str_length) or str.byteslice(str_range); however the substring of +str+ is not allocated as a new string.
In Feature #19314, we concluded that the return value of String#bytesplice
should be changed from the source string to the receiver, because the source
string is useless and confusing when extra arguments are added.
This change should be included in Ruby 3.2.1.
str_enc_copy_direct copies the string encoding over without checking the
frozen status of the string. Because we know that we're safe here (we
only use this function when interpolating strings on the stack via a
concatstrings instruction) we can safely skip this check
This commit adds str_enc_copy_direct, which is like str_enc_copy but
does not check the frozen status of str1 and does not check the validity
of the encoding of str2. This makes certain string operations ~5% faster.
```ruby
puts(Benchmark.measure do
100_000_000.times do
"a".downcase
end
end)
```
Before this patch:
```
7.587598 0.040858 7.628456 ( 7.669022)
```
After this patch:
```
7.133128 0.039809 7.172937 ( 7.183124)
```
The reference updating code for strings is not re-embedding strings
because the code is incorrectly wrapped inside of a
`if (STR_SHARED_P(obj))` clause. Shared strings can't be re-embedded
so this ends up being a no-op. This means that strings can be moved to a
large size pool during compaction, but won't be re-embedded, which would
waste the space.
The following code crashes on my machine:
```
GC.stress = true
str = "testing testing testing"
puts str.capitalize
```
We need to ensure that the object `buffer_anchor` remains on the stack
so it does not get GC'd.
It's questionable whether we want to allow rstrip to work for strings
where the broken coderange occurs before the trailing whitespace and
not after, but this approach is probably simpler, and I don't think
users should expect string operations like rstrip to work on broken
strings.
In some cases, this changes rstrip to raise
Encoding::CompatibilityError instead of ArgumentError. However, as
the problem is related to an encoding issue in the receiver, and due
not due to an issue with an argument, I think
Encoding::CompatibilityError is the more appropriate error.
Fixes [Bug #18931]
During auto-compaction, using length to determine whether or not a
string can be re-embedded may be a problem for newly created strings.
This is because usually it requires a malloc before setting the length.
If the malloc triggers compaction, then the string may be re-embedded
and can cause crashes.
Commit aa2a428 introduced a bug where non-embedded string slices copied
the encoding of the original string. If the original string had a broken
encoding but the slice has valid encoding, then the slice would be
incorrectly marked as broken encoding.
This commit extracts common code between str_substr and rb_str_subseq
into a function called str_subseq.
This commit also applies optimizations in commit 2e88bca to
rb_str_subseq.
str_new_shared already has all the necessary logic to do this
and is also smart enough to skip this step if the source string
is already a shared string itself.
This saves a useless String allocation on each call.
Leave the new coderange unknown if the original encoding is not
ASCII-compatible. Non-ASCII-compatible encoding strings with valid or
broken coderange can end up as ascii-only.
Fixes 9a8f6e392f ("Cheaply derive code range for String#b return
value", 2022-07-25).
Remove the superfluous str_modify_keep_cr() call from rb_str_update().
It ends up calling either rb_str_drop_bytes() or rb_str_splice_0(),
which already does checks if necessary.
The extra call makes the string "independent". This is not always
wanted, in other words, it can keep the same shared root when merely
removing the leading part of a shared string.
This is an inelegant hack, by manually checking for this specific
code point in rb_str_inspect. Some testing indicates that this is
the only code point affected.
It's possible a better fix would be inside of lower-level encoding
code, such that rb_enc_isprint would return false and not true for
codepoint 0x85.
Fixes [Bug #16842]
The result of String#b is a string with an ASCII_8BIT/BINARY encoding. That encoding is ASCII-compatible and has no byte sequences that are invalid for the encoding. If we know the receiver's code range, we can derive the resulting string's code range without needing to perform a full code range scan.
If the RHS has valid encoding, and both strings have the same
encoding, we can use the fast path.
However we need to update the LHS coderange.
```
compare-ruby: ruby 3.2.0dev (2022-07-21T14:46:32Z master cdbb9b8555) [arm64-darwin21]
built-ruby: ruby 3.2.0dev (2022-07-25T07:25:41Z string-concat-vali.. 11a2772bdd) [arm64-darwin21]
warming up...
| |compare-ruby|built-ruby|
|:-------------------|-----------:|---------:|
|binary_concat_7bit | 554.816k| 556.460k|
| | -| 1.00x|
|utf8_concat_7bit | 556.367k| 555.101k|
| | 1.00x| -|
|utf8_concat_UTF8 | 412.555k| 556.824k|
| | -| 1.35x|
```
Previously, it was including one newline when chomp was used,
which is inconsistent with IO#each_line behavior. This makes
behavior consistent with IO#each_line, chomping all paragraph
separators (multiple consecutive newlines), but not single
newlines.
Partially Fixes [Bug #18768]
Not having to fetch the rb_encoding save a significant
amount of time.
Additionally, even when we have to fetch it, we can do
it faster using `ENCODING_GET` rather than `rb_enc_get`.
```
compare-ruby: ruby 3.2.0dev (2022-07-19T08:41:40Z master cb9fd920a3) [arm64-darwin21]
built-ruby: ruby 3.2.0dev (2022-07-21T11:16:16Z faster-buffer-conc.. 4f001f0748) [arm64-darwin21]
warming up...
| |compare-ruby|built-ruby|
|:---------------------|-----------:|---------:|
|binary_concat_utf8 | 510.580k| 565.600k|
| | -| 1.11x|
|binary_concat_binary | 512.653k| 571.483k|
| | -| 1.11x|
|utf8_concat_utf8 | 511.396k| 566.879k|
| | -| 1.11x|
```
If the LHS is ASCII compatible and the RHS is 7BIT
we can directly concat without being concerned about
anything else.
Benchmark:
```
compare-ruby: ruby 3.2.0dev (2022-07-12T15:01:11Z master 71aec68566) [arm64-darwin21]
built-ruby: ruby 3.2.0dev (2022-07-13T10:13:53Z faster-buffer-conc.. a04c10476d) [arm64-darwin21]
warming up...
| |compare-ruby|built-ruby|
|:---------------------|-----------:|---------:|
|binary_append_utf8 | 385.315k| 573.663k|
| | -| 1.49x|
|binary_append_binary | 446.579k| 574.898k|
| | -| 1.29x|
|utf8_append_utf8 | 430.936k| 573.394k|
| | -| 1.33x|
```
Note that in the benchmark, the RHS always have a precomputed
coderange. So the benchmark never enter the slowpath of having to
scan the RHS. However it's extremly likely that we'll end
up scanning it anyway in rb_enc_cr_str_buf_cat
This function was added to a public header in [1] probably
unintentionally since it's not used anywhere, exposes implementation
details, and isn't related to the goals of that pull request.
[1]: 56cc3e99b6
* Add missing space for `String#start_with?`.
* Add missing pluses for `String#tr` and
`Methods for Converting to New String` label.
* Move quote into the tag for `Whitespace in Strings` label.
... only when the message string has a newline.
`p StandardError.new("foo\nbar")` now prints `#<StandardError: "foo\nbar">'
instead of:
#<StandardError:
bar>
[Bug #18170]
Peter Zhu
Nobuyoshi Nakada
Jeremy Evans
Jean Boussier
John Hawthorn
Alan Wu
Burdette Lamar
Kunshan Wang
alexandre184
Matt Valentine-House
Takashi Kokubun
Rômulo Ceccon
John Bampton
Adam Daniels
zverok
Shugo Maeda
S-H-GAMELINKS
Vaevictusnet
Jean Boussier
Kazuki Yamaguchi
Yusuke Endoh
Benoit Daloze
Kevin Menard
Alexander Ilyin