rb_reg_onig_match performs preparation, error handling, and cleanup for
matching a regex against a string. This reduces repetitive code and
removes the need for StringScanner to access internal data of regex.
In `JSON#generate` and `JSON#fast_generate`:
- When the given `opts` is a `JSON::State` the variable is set to
`nil`.
- But it will be never used as the next `if` blocks will not be
executed.
- `JSON::State#configure` does the conversion to `Hash`, the
conversions in the `if` block are just duplication.
- `JSON::State.new` does the same thing with `configure` when an
argument is given.
https://github.com/flori/json/commit/5d9ab87f8e
In `rb_ruby_ripper_parser_allocate`, `r->p` is NULL between creating
`self` and `parser_params` assignment. As GC can happen there, the
typed-data functions for it need to consider the case.
OpenSSL::Cipher#pkcs5_keyivgen
(https://github.com/ruby/openssl/pull/647)
OpenSSL::Cipher#pkcs5_keyivgen should only be used when it is
absolutely necessary for compatibility with ancient applications.
Having an example can be misleading. We already have another example
for OpenSSL::Cipher in which PBKDF2 is used to derive a key.
As described in the rdoc of OpenSSL::Cipher#pkcs5_keyivgen, it is
compatible with PKCS#5 PBES1 (PKCS#5 v1.5) only when used in combination
of a hash function MD2, MD5, or SHA-1, and a cipher DES-CBC or RC2-CBC.
This example uses MD5 as the hash function and combines it with AES.
This is considered insecure and also using a non-standard technique to
derive longer keys.
https://github.com/ruby/openssl/commit/e379cc0cca
(https://github.com/ruby/openssl/pull/646)
Add OpenSSL::PKey.new_raw_private_key, #raw_private_key and public
equivalents. These methods are useful for importing and exporting keys
that support "raw private/public key". Currently, OpenSSL implements
X25519/X448 and Ed25519/Ed448 keys.
[rhe: rewrote commit message]
https://github.com/ruby/openssl/commit/3f29525618
Co-authored-by: Bart de Water <bartdewater@gmail.com>
According to the C99 specification section 7.20.3.2 paragraph 2:
> If ptr is a null pointer, no action occurs.
So we do not need to check that the pointer is a null pointer.
The following script leaks memory in Ripper:
```ruby
require "ripper"
20.times do
100_000.times do
Ripper.parse("")
end
puts `ps -o rss= -p #{$$}`
end
```
(https://github.com/ruby/fiddle/pull/123)
This commit adds two new methods, `Fiddle::Pointer.read` and
`Fiddle::Pointer.write`. Both methods take an address, and will read or
write bytes at that address respectively.
For example we can read from an address without making a Pointer object:
```ruby
Fiddle::Pointer.read(address, 5) # read 5 bytes
```
We can also write to an address without allocating a Pointer object:
```ruby
Fiddle::Pointer.write(address, "bytes") # write 5 bytes
```
This allows us to read / write memory at arbitrary addresses without
instantiating a new `Fiddle::Pointer` object.
Examples where this API would be useful
[1](f03481d28b/lib/tenderjit/fiddle_hacks.rb (L26-L28))
[2](77c8daa2d4/lib/ruby_vm/rjit/c_pointer.rb (L193))
[3](77c8daa2d4/lib/ruby_vm/rjit/c_pointer.rb (L284))
I also added a writer method for the same reasons as the reader.
---------
https://github.com/ruby/fiddle/commit/04238cefed
Co-authored-by: Sutou Kouhei <kou@clear-code.com>
clang generates a warning:
../../../../ext/openssl/ossl_pkey.c:326:22: error: implicit truncation from 'int' to a one-bit wide bit-field changes value from 1 to -1 [-Werror,-Wsingle-bit-bitfield-constant-conversion]
arg->interrupted = 1;
^ ~
1 error generated.
https://github.com/ruby/openssl/commit/4b2ba7b022
RUBY_OPENSSL_EXTCFLAGS and RUBY_OPENSSL_EXTLDFLAGS have been added for
the primary purpose of appending custom warning flags during
development and CI.
Since checking programs generated by mkmf may not be completely
warning-free, we don't want to apply -Werror that may be supplied from
those environment variables.
https://github.com/ruby/openssl/commit/2a95b971d5
This commit is a workaround to avoid the error below that the
`OpenSSL::PKey.read` fails with the OpenSSL 3.0 FIPS mode.
```
$ openssl genrsa -out key.pem 4096
$ ruby -e "require 'openssl'; OpenSSL::PKey.read(File.read('key.pem'))"
-e:1:in `read': Could not parse PKey (OpenSSL::PKey::PKeyError)
from -e:1:in `<main>'
```
The root cause is on the OpenSSL side. The `OSSL_DECODER_CTX_set_selection`
doesn't apply the selection value properly if there are multiple providers, and
a provider (e.g. "base" provider) handles the decoder implementation, and
another provider (e.g. "fips" provider) handles the keys.
The workaround is to create `OSSL_DECODER_CTX` variable each time without using
the `OSSL_DECODER_CTX_set_selection`.
https://github.com/ruby/openssl/commit/5ff4a31621
The vast majority have no reference so it's just a matter of setting the flags.
For the couple exception, they have very little references so it's
easy.
https://github.com/ruby/openssl/commit/2c7c6de69e
According to the `mkmf.rb#init_mkmf`, there are command line options below.
* `--with-cflags` to set the `cflags`
* `--with-ldflags` to set the `ldflags`
For example the following command compiles with the specified flags. Note that
`MAKEFLAGS` is to print the compiler command lines.
```
$ MAKEFLAGS="V=1" \
bundle exec rake compile -- \
--with-cflags="-Wundef -Werror" \
--with-ldflags="-fstack-protector"
```
However, I couldn't find command line options to append the flags. And this
commit is to append the `cflags` and `ldflags` by the environment variables.
```
$ MAKEFLAGS="V=1" \
RUBY_OPENSSL_EXTCFLAGS="-Wundef -Werror" \
RUBY_OPENSSL_EXTLDFLAGS="-fstack-protector" \
bundle exec rake compile
```
https://github.com/ruby/openssl/commit/b551eb86f6
Introduce Universal Parser mode for the parser.
This commit includes these changes:
* Introduce `UNIVERSAL_PARSER` macro. All of CRuby related functions
are passed via `struct rb_parser_config_struct` when this macro is enabled.
* Add CI task with 'cppflags=-DUNIVERSAL_PARSER' for ubuntu.
* Unify length field for embedded and heap strings
The length field is of the same type and position in RString for both
embedded and heap allocated strings, so we can unify it.
* Remove RSTRING_EMBED_LEN
This reverts commit 10621f7cb9.
This was reverted because the gc integrity build started failing. We
have figured out a fix so I'm reopening the PR.
Original commit message:
Fix cvar caching when class is cloned
The class variable cache that was added in
ruby#4544 changed the behavior of class
variables on cloned classes. As reported when a class is cloned AND a
class variable was set, and the class variable was read from the
original class, reading a class variable from the cloned class would
return the value from the original class.
This was happening because the IC (inline cache) is stored on the ISEQ
which is shared between the original and cloned class, therefore they
share the cache too.
To fix this we are now storing the `cref` in the cache so that we can
check if it's equal to the current `cref`. If it's different we don't
want to read from the cache. If it's the same we do. Cloned classes
don't share the same cref with their original class.
This will need to be backported to 3.1 in addition to 3.2 since the bug
exists in both versions.
We also added a marking function which was missing.
Fixes [Bug #19379]
Co-authored-by: Aaron Patterson <tenderlove@ruby-lang.org>
The class variable cache that was added in
https://github.com/ruby/ruby/pull/4544 changed the behavior of class
variables on cloned classes. As reported when a class is cloned AND a
class variable was set, and the class variable was read from the
original class, reading a class variable from the cloned class would
return the value from the original class.
This was happening because the IC (inline cache) is stored on the ISEQ
which is shared between the original and cloned class, therefore they
share the cache too.
To fix this we are now storing the `cref` in the cache so that we can
check if it's equal to the current `cref`. If it's different we don't
want to read from the cache. If it's the same we do. Cloned classes
don't share the same cref with their original class.
This will need to be backported to 3.1 in addition to 3.2 since the bug
exists in both versions.
We also added a marking function which was missing.
Fixes [Bug #19379]
Co-authored-by: Aaron Patterson <tenderlove@ruby-lang.org>
This reverts commit 35da41b29b.
This updates `io-console` to avoid accessing the internal details of `rb_io_t`
and instead use `rb_io_descriptor` and `rb_io_path` etc.
* Add rb_io_path and rb_io_open_descriptor.
* Use rb_io_open_descriptor to create PTY objects
* Rename FMODE_PREP -> FMODE_EXTERNAL and expose it
FMODE_PREP I believe refers to the concept of a "pre-prepared" file, but
FMODE_EXTERNAL is clearer about what the file descriptor represents and
aligns with language in the IO::Buffer module.
* Ensure that rb_io_open_descriptor closes the FD if it fails
If FMODE_EXTERNAL is not set, then it's guaranteed that Ruby will be
responsible for closing your file, eventually, if you pass it to
rb_io_open_descriptor, even if it raises an exception.
* Rename IS_EXTERNAL_FD -> RUBY_IO_EXTERNAL_P
* Expose `rb_io_closed_p`.
* Add `rb_io_mode` to get IO mode.
---------
Co-authored-by: KJ Tsanaktsidis <ktsanaktsidis@zendesk.com>
The C extension defines an `included` hook for the
`JSON::Ext::Generator::GeneratorMethods::String` module but neglects to
call `super` in the hook. This can break the functionality of various
other code that rely on the fact that `included` on `Module` will always
be called.
https://github.com/flori/json/commit/cd8bbe56a3
The commit <https://github.com/ruby/openssl/commit/c5b2bc1268bc> made the warnings below
in the case of OpenSSL 1.1 where the `OPENSSL_FIPS` macro is not defined.
```
$ bundle install --standalone
$ bundle exec rake compile -- \
--with-openssl-dir=$HOME/.local/openssl-1.1.1t-debug \
--with-cflags="-Wundef"
mkdir -p tmp/x86_64-linux/openssl/3.2.1
cd tmp/x86_64-linux/openssl/3.2.1
/usr/local/ruby-3.2.1/bin/ruby -I. -r.rake-compiler-siteconf.rb ../../../../ext/openssl/extconf.rb -- --with-openssl-dir=/home/jaruga/.local/openssl-1.1.1t-debug --with-cflags=-Wundef
...
gcc -I. -I/usr/local/ruby-3.2.1/include/ruby-3.2.0/x86_64-linux -I/usr/local/ruby-3.2.1/include/ruby-3.2.0/ruby/backward -I/usr/local/ruby-3.2.1/include/ruby-3.2.0 -I../../../../ext/openssl -DRUBY_EXTCONF_H=\"extconf.h\" -I/home/jaruga/.local/openssl-1.1.1t-debug/include -fPIC -Wundef -o ossl.o -c ../../../../ext/openssl/ossl.c
../../../../ext/openssl/ossl.c: In function ‘ossl_fips_mode_get’:
../../../../ext/openssl/ossl.c:425:7: warning: "OPENSSL_FIPS" is not defined, evaluates to 0 [-Wundef]
425 | #elif OPENSSL_FIPS
| ^~~~~~~~~~~~
../../../../ext/openssl/ossl.c: In function ‘ossl_fips_mode_set’:
../../../../ext/openssl/ossl.c:460:7: warning: "OPENSSL_FIPS" is not defined, evaluates to 0 [-Wundef]
460 | #elif OPENSSL_FIPS
| ^~~~~~~~~~~~
../../../../ext/openssl/ossl.c: In function ‘Init_openssl’:
../../../../ext/openssl/ossl.c:1218:7: warning: "OPENSSL_FIPS" is not defined, evaluates to 0 [-Wundef]
1218 | #elif OPENSSL_FIPS
| ^~~~~~~~~~~~
...
cp tmp/x86_64-linux/openssl/3.2.1/openssl.so tmp/x86_64-linux/stage/lib/openssl.so
```
https://github.com/ruby/openssl/commit/b4228cbcd6
This commit is to implement the `OpenSSL::OPENSSL_FIPS`, `ossl_fips_mode_get`
and `ossl_fips_mode_set` to pass the test `test/openssl/test_fips.rb`.
It seems that the `OPENSSL_FIPS` macro is not used on the FIPS mode case any
more, and some FIPS related APIs also were removed in OpenSSL 3.
See the document <https://github.com/openssl/openssl/blob/master/doc/man7/migration_guide.pod#removed-fips_mode-and-fips_mode_set>
the section OPENSSL 3.0 > Main Changes from OpenSSL 1.1.1 >
Other notable deprecations and changes - Removed FIPS_mode() and FIPS_mode_set() .
The `OpenSSL::OPENSSL_FIPS` returns always true in OpenSSL 3 because the used
functions `EVP_default_properties_enable_fips` and `EVP_default_properties_is_fips_enabled`
works with the OpenSSL installed without FIPS option.
The `TEST_RUBY_OPENSSL_FIPS_ENABLED` is set on the FIPS mode case on the CI.
Because I want to test that the `OpenSSL.fips_mode` returns the `true` or
'false' surely in the CI. You can test the FIPS mode case by setting
`TEST_RUBY_OPENSSL_FIPS_ENABLED` on local too. Right now I don't find a better
way to get the status of the FIPS mode enabled or disabled for this purpose. I
am afraid of the possibility that the FIPS test case is unintentionally skipped.
I also replaced the ambiguous "returns" with "should return" in the tests.
https://github.com/ruby/openssl/commit/c5b2bc1268
Checking by `__STDC_VERSION__` is unreliable because old gcc 4.8
supports `-std=gnu11` option but does not implement `_Thread_local`.
Check the implementation directly instead.
The socket extensions rubysocket.h pulls in the "private" include/gc.h,
which now depends on vm_core.h. vm_core.h pulls in id.h
when tool/update-deps generates the dependencies for the makefiles, it
generates the line for id.h to be based on VPATH, which is configured in
the extconf.rb for each of the extensions. By default VPATH does not
include the actual source directory of the current Ruby so the
dependency fails to resolve and linking fails.
We need to append the topdir and top_srcdir to VPATH to have the
dependancy picked up correctly (and I believe we need both of these to
cope with in-tree and out-of-tree builds).
I copied this from the approach taken in
https://github.com/ruby/ruby/blob/master/ext/objspace/extconf.rb#L3
Remove !USE_RVARGC code
[Feature #19579]
The Variable Width Allocation feature was turned on by default in Ruby
3.2. Since then, we haven't received bug reports or backports to the
non-Variable Width Allocation code paths, so we assume that nobody is
using it. We also don't plan on maintaining the non-Variable Width
Allocation code, so we are going to remove it.
JRuby has its own implementation of the `openssl` library in
jruby-openssl. The simplest way for us to allow users to set
openssl as a gem dependency is to ship a stub gem that just
depends on jruby-openssl. This patch adds that to the gemspec.
Additional work may be required to fit this stub gem into the test
and release process.
See #20 for more details.
https://github.com/ruby/openssl/commit/74ccaa5e18
The dependency of extconf.h on bigdecimal.gemspec does not make sense
as far as no rule is defined for it. Also, the relationship between
extension library and gemspec file is various in default gems, and
does not work well.
https://github.com/ruby/bigdecimal/commit/7f99b28552
* remove false call-seq (output from Ruby parsing is cleaner)
* explain output: argument in plain words
* change parameter name in docs of #dump_shapes (typo)
JRuby currently ships its own internal bigdecimal extension as
part of the core libraries. In order for users to be able to add
bigdecimal to their Gemfile or gem dependencies, we need to stub
out the C extension and just load the extension shipped with
JRuby.
In the future we will try to move our BigDecimal implementation
into the gem, but for now this is the simplest way to make it
installable on JRuby.
See #169https://github.com/ruby/bigdecimal/commit/829956c643
Right now the attached object is stored as an instance variable
and all the call sites that either get or set it have to know how it's
stored.
It's preferable to hide this implementation detail behind accessors
so that it is easier to change how it's stored.
This version primarily updates the JRuby extension to use
SnakeYAML Engine, a newer version of the SnakeYAML library, which
also updates YAML support to 1.2. The JRuby extension now also
exposes settings for the parser.
https://github.com/ruby/psych/commit/6f2b16b343
See jruby/jruby#7570 for some of the justification for this move. We only
require the parser from SnakeYAML, but in the original form it is
encumbered with Java object serialization code that keeps getting
flagged as a CVE risk. We disagree with the assessment, at least
as it pertains to JRuby (we do not use the code in question) but
our inclusion of the library continues to get flagged by auditing
tools.
This commit starts the process of moving to the successor library,
SnakeYAML Engine. The parser API is largely unchanged, except as
seen in this commit. No Java exceptions are thrown, but a number
of Psych tests fail (possibly due to Engine being YAML 1.2 only).
(https://github.com/ruby/fiddle/pull/119)
The documentation for `Fiddle.dlwrap` and `Fiddle.dlunwrap` were not
very accurate and pretty confusing. This commit updates the
documentation so it's easier to understand what the methods do.
ObjectSpace.trace_object_allocations_start could crash since it adds a
TracePoint for when objects are freed. However, TracePoint could crash
since it modifies st tables while inside the GC that is trying to free
the object. This could cause a memory allocation to happen which would
crash if it triggers another GC.
See a crash log: http://ci.rvm.jp/results/trunk@ruby-sp1/4373707
(https://github.com/ruby/fiddle/pull/118)
String#unpack1 is available since Ruby 2.4, and support for older than
Ruby 2.5 was dropped by #85.
Also simplified a common return statement.
The behavior of EVP_PKEY_public_check changed between OpenSSL 1.1.1
and 3.0 so that it no longer validates the private key. Instead, private
keys can be validated through EVP_PKEY_private_check and
EVP_PKEY_pairwise_check.
[ky: simplified condition to use either EVP_PKEY_check() or
EVP_PKEY_public_check().]
https://github.com/ruby/openssl/commit/e38a63ab3d
This fixes a linkage error about `ossl_ssl_type` on platforms which do
not have socket, like WASI.
Even before this patch, some items are disabled under `OPENSSL_NO_SOCK` since
ee22fad45d
However, due to some new use of OpenSSL::SSL::Socket over the past few years,
the build under `OPENSSL_NO_SOCK` had been broken.
This patch guards whole `OpenSSL::SSL` items by `OPENSSL_NO_SOCK`.
[ky: adjusted to apply on top of my previous commit that removed the
OpenSSL::ExtConfig, and added a guard to lib/openssl/ssl.rb.]
https://github.com/ruby/openssl/commit/b0cfac6a96
As noted in commit https://github.com/ruby/openssl/commit/a2ed156cc9f1 ("test/test_ssl: do not run NPN tests
for LibreSSL >= 2.6.1", 2017-08-13), NPN is known not to work properly
on LibreSSL.
Disable NPN support on LibreSSL, whether OPENSSL_NO_NEXTPROTONEG is
defined or not.
NPN is less relevant today anyway. Let's also silence test suite when
it's not available.
https://github.com/ruby/openssl/commit/289f6e0e1f
When an object becomes "too complex" (in other words it has too many
variations in the shape tree), we transition it to use a "too complex"
shape and use a hash for storing instance variables.
Without this patch, there were rare cases where shape tree growth could
"explode" and cause performance degradation on what would otherwise have
been cached fast paths.
This patch puts a limit on shape tree growth, and gracefully degrades in
the rare case where there could be a factorial growth in the shape tree.
For example:
```ruby
class NG; end
HUGE_NUMBER.times do
NG.new.instance_variable_set(:"@unique_ivar_#{_1}", 1)
end
```
We consider objects to be "too complex" when the object's class has more
than SHAPE_MAX_VARIATIONS (currently 8) leaf nodes in the shape tree and
the object introduces a new variation (a new leaf node) associated with
that class.
For example, new variations on instances of the following class would be
considered "too complex" because those instances create more than 8
leaves in the shape tree:
```ruby
class Foo; end
9.times { Foo.new.instance_variable_set(":@uniq_#{_1}", 1) }
```
However, the following class is *not* too complex because it only has
one leaf in the shape tree:
```ruby
class Foo
def initialize
@a = @b = @c = @d = @e = @f = @g = @h = @i = nil
end
end
9.times { Foo.new }
``
This case is rare, so we don't expect this change to impact performance
of most applications, but it needs to be handled.
Co-Authored-By: Aaron Patterson <tenderlove@ruby-lang.org>
Count how many "variations" each class creates. A "variation" is a a
unique ordering of instance variables on a particular class. This can
also be thought of as a branch in the shape tree.
For example, the following Foo class will have 2 variations:
```ruby
class Foo ; end
Foo.new.instance_variable_set(:@a, 1) # case 1: creates one variation
Foo.new.instance_variable_set(:@b, 1) # case 2: creates another variation
foo = Foo.new
foo.instance_variable_set(:@a, 1) # does not create a new variation
foo.instance_variable_set(:@b, 1) # does not create a new variation (a continuation of the variation in case 1)
```
We will use this number to limit the amount of shapes that a class can
create and fallback to using a hash iv lookup.
Co-Authored-By: Aaron Patterson <tenderlove@ruby-lang.org>
LibreSSL 3.4 added EVP_DigestSign() and EVP_DigestVerify(). Use them
when available to prepare for the addition of Ed25519 support in
LibreSSL 3.7.
https://github.com/ruby/openssl/commit/475b2bf766
I see several arguments in doing so.
First they use a non trivial amount of memory, so for various memory
profiling/mapping tools it is relevant to have visibility of the space
occupied by shapes.
Then, some pathological code can create a tons of shape, so it is
valuable to have a way to have a way to observe shapes without having
to compile Ruby with `SHAPE_DEBUG=1`.
And additionally it's likely much faster to dump then this way than
to use `RubyVM::Shape`.
There are however a few open questions:
- Shapes can't respect the `since:` argument. Not sure what to do when
it is provided. Would probably make sense to not dump them.
- Maybe it would make more sense to have a separate `ObjectSpace.dump_shapes`?
- Maybe instead `dump_all` should take a `shapes: false` argument?
Additionally, `ObjectSpace.dump_shapes` is added for the use case of
debugging the evolution of the shape tree.
Sutou Kouhei
Peter Zhu
Benoit Daloze
Nobuyoshi Nakada
Hiroshi SHIBATA
Dimitar Haralanov
yui-knk
Kazuki Yamaguchi
Ryo Kajiwara
cryptogopher
Jean byroot Boussier
Aaron Patterson
qwyng
Jun Aruga
Jean Boussier
Samuel Williams
eileencodes
NARUSE, Yui
Ufuk Kayserilioglu
Akinori MUSHA
HoNooD
Matt Valentine-House
BurdetteLamar
Charles Oliver Nutter
OKURA Masafumi
zverok
Maciej Rzasa
Jean Boussier
Mau Magnaguagno
Khem Raj
Yusuke Nakamura
Kazuhiro NISHIYAMA
Rick Hull
Lars Kanis
dsisnero
Kevin Murphy
Marco Costa
Johnny Willemsen
Joe Truba
Yuta Saito
Jemma Issroff
MSP-Greg
Henrique Bontempo
Theo Buehler
Daniel Colson
S-H-GAMELINKS