After a pthread for getaddrinfo is detached, we cannot predict when the
thread will exit. It would lead to a segfault by setting
pthread_setaffinity to the terminated pthread. I guess this problem
would be more likely to occur in high-load environments.
This change detaches the pthread after pthread_setaffinity is called.
[Feature #19965]
Add a new API rb_profile_thread_frames(), which is essentialy a
per-thread version of rb_profile_frames().
While the original rb_profile_frames() always returns results about the
current active thread obtained by GET_EC(), this new API takes a Thread
to be profiled as an argument.
This should come in handy when profiling I/O-bound programs such as
webapps, since this new API allows us to learn about Threads performing
I/O (which do not have the GVL).
Profiling worker threads (such as Sidekiq workers) may be another
application.
Implements [Feature #10602]
Co-authored-by: Mike Perham <mike@perham.net>
extconf.rb
(https://github.com/ruby/zlib/pull/69)
The android NDK (android-ndk-r21e) does not have crc32_z, adler32_z, nor
z_size_t in its zlib.h header file. However, it _does_ have the crc32_z
and adler32_z symbols in the libz.a static library!
mkmf performs two tests for have_func:
* It sees if a program that includes the header and takes the address of
the symbol can compile
* It sees if a program that defines the symbol as `extern void sym_name()`
and calls it can be linked
If either test works, it considers the function present. The
android-ndk-r21e is passing the second test but not the first for
crc32_z/adler32_z. So, we define HAVE_ZLIB_SIZE_T_FUNCS, but then can't
actually compile the extension (since the prototypes aren't in the
header file).
We can keep this working how it was working before by _also_ checking
for `have_type("z_size_t", "zlib.h")`. The have_type check _only_ looks
in the header file for the type; if a program including the header file
and using the type can't compile, the type is considered absent
regardless of what might be in libz.a.
https://github.com/ruby/zlib/commit/3b9fe962d8
We use the Cloudflare fork of zlib
(https://github.com/cloudflare/zlib), which we find gives improved
performance on AWS Graviton ARM instances. That fork does not define
crc32_z and alder32_z functions.
Until two days ago, Ruby's zlib gem worked fine, because cloudflare zlib
_also_ did not define z_size_t, which meant Ruby did not try and use
these functions.
Since a3ba99596d
however, cloudflare zlib _does_ define z_size_t (but NOT crc32_z or
alder32_z). The zlib gem would try and use these nonexistant
functions and not compile.
This patch fixes it by actually specifically detecting the functions
that the gem wants to call, rather than just the presence of the
z_size_t type.
https://github.com/ruby/zlib/commit/c96e8b9a57
When pthread_create is available, rb_getaddrinfo creates a pthread and
executes getaddrinfo(3) in it. The caller thread waits for the pthread
to complete, but detaches it if interrupted. This allows name resolution
to be interuppted by Timeout.timeout, etc. even if it takes a long time
(for example, when the DNS server does not respond). [Feature #19965]
I don't think it's possible to create a CI with a mid which would need
escaping to be in a JSON string, but I think we might as well not rely
on that assumption.
All kind of AST nodes use same struct RNode, which has u1, u2, u3 union members
for holding different kind of data.
This has two problems.
1. Low flexibility of data structure
Some nodes, for example NODE_TRUE, don’t use u1, u2, u3. On the other hand,
NODE_OP_ASGN2 needs more than three union members. However they use same
structure definition, need to allocate three union members for NODE_TRUE and
need to separate NODE_OP_ASGN2 into another node.
This change removes the restriction so make it possible to
change data structure by each node type.
2. No compile time check for union member access
It’s developer’s responsibility for using correct member for each node type when it’s union.
This change clarifies which node has which type of fields and enables compile time check.
This commit also changes node_buffer_elem_struct buf management to handle
different size data with alignment.
* Reword the description in README for more clarity.
* Add a compatibility matrix of our stable branches and explain the
maintenance policy.
* Remove the obsolete paragraph for how to use the gem in Ruby 2.3,
which is no longer supported.
https://github.com/ruby/openssl/commit/7691034fcb
instead of looking of NIDs and then using X509V3_EXT_nconf_nid,
instead just pass strings to X509V3_EXT_nconf, which has all the logic for
processing dealing with generic extensions
also process the oid through ln2nid() to retain compatibility.
[rhe: tweaked commit message and added a test case]
https://github.com/ruby/openssl/commit/9f15741331
In TLS 1.2 or before, if DH group parameters for DHE are not supplied
with SSLContext#tmp_dh= or #tmp_dh_callback=, we currently use the
self-generated parameters added in commit https://github.com/ruby/openssl/commit/bb3399a61c03 ("support 2048
bit length DH-key", 2016-01-15) as the fallback.
While there is no known weakness in the current parameters, it would be
a good idea to switch to pre-defined, more well audited parameters.
This also allows the fallback to work in the FIPS mode.
The PEM encoding was derived with:
# RFC 7919 Appendix A.1. ffdhe2048
print OpenSSL::PKey.read(OpenSSL::ASN1::Sequence([OpenSSL::ASN1::Integer((<<-END).split.join.to_i(16)), OpenSSL::ASN1::Integer(2)]).to_der).to_pem
FFFFFFFF FFFFFFFF ADF85458 A2BB4A9A AFDC5620 273D3CF1
D8B9C583 CE2D3695 A9E13641 146433FB CC939DCE 249B3EF9
7D2FE363 630C75D8 F681B202 AEC4617A D3DF1ED5 D5FD6561
2433F51F 5F066ED0 85636555 3DED1AF3 B557135E 7F57C935
984F0C70 E0E68B77 E2A689DA F3EFE872 1DF158A1 36ADE735
30ACCA4F 483A797A BC0AB182 B324FB61 D108A94B B2C8E3FB
B96ADAB7 60D7F468 1D4F42A3 DE394DF4 AE56EDE7 6372BB19
0B07A7C8 EE0A6D70 9E02FCE1 CDF7E2EC C03404CD 28342F61
9172FE9C E98583FF 8E4F1232 EEF28183 C3FE3B1B 4C6FAD73
3BB5FCBC 2EC22005 C58EF183 7D1683B2 C6F34A26 C1B2EFFA
886B4238 61285C97 FFFFFFFF FFFFFFFF
END
https://github.com/ruby/openssl/commit/a5527cb4f4
Remove the OSSL_DEBUG flag and OpenSSL.mem_check_start which is only
compiled when the flag is given. They are meant purely for development
of Ruby/OpenSSL.
OpenSSL.mem_check_start helped us find memory leak bugs in past, but
it is no longer working with the recent OpenSSL versions. Let's just
remove it now.
https://github.com/ruby/openssl/commit/8c7a6a17e2
[Bug #19012]
man recvmsg(2) states:
> Return Value
> These calls return the number of bytes received, or -1 if an error occurred.
> The return value will be 0 when the peer has performed an orderly shutdown.
Not too sure how one is supposed to make the difference between a packet of
size 0 and a closed connection.
This is a workaround for the decoding issue in ossl_pkey_read_generic().
The issue happens in the case that a key management provider is different from
a decoding provider.
Try all the non-zero selections in order, instead of selection 0 for OpenSSL 3
to avoid the issue.
https://github.com/ruby/openssl/commit/db688fa739
* Fix the wrong man reference.
* According to the LIBRESSL_VERSION_NUMBER(3), the value always ends with 00f.
```
$ man -M /home/jaruga/.local/libressl-6650dce/share/man/ 3 LIBRESSL_VERSION_NUMBER
...
DESCRIPTION
OPENSSL_VERSION_NUMBER and LIBRESSL_VERSION_NUMBER are numeric release version
identifiers. The first two digits contain the major release number, the third and
fourth digits the minor release number, and the fifth and sixth digits the fix re‐
lease number. For OpenSSL, the seventh and eight digits contain the patch release
number and the final digit is 0 for development, 1 to e for betas 1 to 14, or f
for release. For LibreSSL, OPENSSL_VERSION_NUMBER is always 0x020000000, and
LIBRESSL_VERSION_NUMBER always ends with 00f.
```
https://github.com/ruby/openssl/commit/296c859d18
* Updated the `OpenSSL::OPENSSL_VERSION_NUMBER` comment explaining the format.
* Added the `OpenSSL::LIBRESSL_VERSION_NUMBER` to print LibreSSL version number,
in the case that Ruby OpenSSL binding is compiled with LibreSSL. Note
`test/openssl/utils.rb#libressl?` is not using this value in it for now.
* Update `rake debug` to print the values in a readable way, adding
`OpenSSL::OPENSSL_VERSION_NUMBER` and `OpenSSL::LIBRESSL_VERSION_NUMBER`.
https://github.com/ruby/openssl/commit/d19e6360ed
Error entries in the OpenSSL error queue may contain additional
contextual information associated with the error, which can be helpful
when debugging.
This "additional data" is currently only printed to stderr when
OpenSSL.debug is enabled. Let's include this in the exception messages
raised with ossl_raise(), too.
$ ruby -Ilib -ropenssl -e'OpenSSL.debug=true; OpenSSL::SSL::SSLContext.new.ecdh_curves="P-256:not-a-curve"'
-e:1: warning: error on stack: error:0A080106:SSL routines:gid_cb:passed invalid argument (group 'not-a-curve' cannot be set)
-e:1:in `ecdh_curves=': passed invalid argument (group 'not-a-curve' cannot be set) (OpenSSL::SSL::SSLError)
from -e:1:in `<main>'
https://github.com/ruby/openssl/commit/1c5bbdd68e
When compiled with OpenSSL <= 1.1.1, OpenSSL::SSL::SSLContext#setup
does not raise an exception on an error return from
SSL_CTX_load_verify_locations(), but instead only prints a verbose-mode
warning. This is not helpful since it very likely indicates an actual
error, such as the specified file not being readable.
Also, OpenSSL's error queue is not correctly cleared:
$ ruby -w -ropenssl -e'OpenSSL.debug=true; ctx=OpenSSL::SSL::SSLContext.new; ctx.ca_file="bad-path"; ctx.setup; pp OpenSSL.errors'
-e:1: warning: can't set verify locations
["error:02001002:system library:fopen:No such file or directory",
"error:2006D080:BIO routines:BIO_new_file:no such file",
"error:0B084002:x509 certificate routines:X509_load_cert_crl_file: system lib"]
The behavior is currently different when compiled with OpenSSL >= 3.0:
SSLError is raised if SSL_CTX_load_verify_file() or
SSL_CTX_load_verify_dir() fails.
This inconsistency was unintentionally introduced by commit https://github.com/ruby/openssl/commit/5375a55ffc35
("ssl: use SSL_CTX_load_verify_{file,dir}() if available", 2020-02-22).
However, raising SSLError seems more appropriate in this situation.
Let's adjust the OpenSSL <= 1.1.1 code so that it behaves the same way
as the OpenSSL >= 3.0 code currently does.
Fixes: https://github.com/ruby/openssl/issues/649https://github.com/ruby/openssl/commit/7eb10f7b75
OpenSSL built from the source creates the library directory to the
`/path/to/openssl_dir/lib64` as a default.
In the case, the `bundle exec rake compile -- --with-openssl-dir=<openssl_dir>`
cannot compile with the lib64 directory, and may compile with system OpenSSL's
libraries unintentionally. This commit is to check this case to avoid linking
with an unintentional library directory.
https://github.com/ruby/openssl/commit/ca54087462
Describe the behavior of OpenSSL::PKey::{DH,DSA,EC,RSA}#to_pem
and #to_der more clearly. They return a different result depending on
whether the pkey is a public or private key. This was not documented
adequately.
Also, suggest the use of OpenSSL::PKey::PKey#private_to_pem
and #public_to_pem instead, if possible.
https://github.com/ruby/openssl/commit/d22769af8f
Suggest the use of OpenSSL::PKey::PKey#private_to_pem and #public_to_pem
in the top-level documentation. For new programs, these are recommended
over OpenSSL::PKey::RSA#export (also aliased as #to_s and #to_pem)
unless there is a specific reason to use it, i.e., unless the PKCS#1
output format specifically is required.
The output format of OpenSSL::PKey::RSA#export depends on whether the
key is a public key or a private key, which is very counter-intuitive.
Additionally, when called with arguments to encrypt a private key, as in
this example, OpenSSL's own, non-standard format is used. The man page
of PEM_write_bio_PrivateKey_traditional(3) in OpenSSL 1.1.1 or later
states that it "should only be used for compatibility with legacy
programs".
https://github.com/ruby/openssl/commit/56312038d6
Let's consistently use the word "password". Although they are considered
synonymous, the mixed usage in the rdoc can cause confusion.
OpenSSL::KDF.scrypt is an exception. This is because RFC 7914 refers to
the input parameter as "passphrase".
https://github.com/ruby/openssl/commit/06d67640e9
Enrich SSLError's message with the low-level certificate verification
result, even if SSL_get_error() returns SSL_ERROR_SYSCALL. This is
currently done on SSL_ERROR_SSL only.
According to the man page of SSL_get_error(), SSL_ERROR_SYSCALL may be
returned for "other errors, check the error queue for details". This
apparently means we have to treat SSL_ERROR_SYSCALL, if errno is not
set, as equivalent to SSL_ERROR_SSL.
https://github.com/ruby/openssl/commit/5113777e82
(https://github.com/ruby/zlib/pull/55)
zstream_discard_input was encoding and character-aware when given input is user-provided, so this discards `len` chars instead of `len` bytes.
Also Zlib.gunzip explains in its rdoc that it is equivalent with the following code, but this doesn't fail for UTF-8 String.
```ruby
string = %w[1f8b0800c28000000003cb48cdc9c9070086a6103605000000].pack("H*").force_encoding('UTF-8')
sio = StringIO.new(string)
p gz.read #=> "hello"
gz&.close
p Zlib.gunzip(string) #=> Zlib::DataError
```
Reported and discovered by eagletmt at https://twitter.com/eagletmt/status/1689692467929694209https://github.com/ruby/zlib/commit/c5e58bc62a
Both for being closer to real IOs and also because it's a convenient API
in multithreaded scenarios.
Co-authored-by: Jean Boussier <jean.boussier@gmail.com>
rb_reg_onig_match performs preparation, error handling, and cleanup for
matching a regex against a string. This reduces repetitive code and
removes the need for StringScanner to access internal data of regex.
In `JSON#generate` and `JSON#fast_generate`:
- When the given `opts` is a `JSON::State` the variable is set to
`nil`.
- But it will be never used as the next `if` blocks will not be
executed.
- `JSON::State#configure` does the conversion to `Hash`, the
conversions in the `if` block are just duplication.
- `JSON::State.new` does the same thing with `configure` when an
argument is given.
https://github.com/flori/json/commit/5d9ab87f8e
In `rb_ruby_ripper_parser_allocate`, `r->p` is NULL between creating
`self` and `parser_params` assignment. As GC can happen there, the
typed-data functions for it need to consider the case.
OpenSSL::Cipher#pkcs5_keyivgen
(https://github.com/ruby/openssl/pull/647)
OpenSSL::Cipher#pkcs5_keyivgen should only be used when it is
absolutely necessary for compatibility with ancient applications.
Having an example can be misleading. We already have another example
for OpenSSL::Cipher in which PBKDF2 is used to derive a key.
As described in the rdoc of OpenSSL::Cipher#pkcs5_keyivgen, it is
compatible with PKCS#5 PBES1 (PKCS#5 v1.5) only when used in combination
of a hash function MD2, MD5, or SHA-1, and a cipher DES-CBC or RC2-CBC.
This example uses MD5 as the hash function and combines it with AES.
This is considered insecure and also using a non-standard technique to
derive longer keys.
https://github.com/ruby/openssl/commit/e379cc0cca
(https://github.com/ruby/openssl/pull/646)
Add OpenSSL::PKey.new_raw_private_key, #raw_private_key and public
equivalents. These methods are useful for importing and exporting keys
that support "raw private/public key". Currently, OpenSSL implements
X25519/X448 and Ed25519/Ed448 keys.
[rhe: rewrote commit message]
https://github.com/ruby/openssl/commit/3f29525618
Co-authored-by: Bart de Water <bartdewater@gmail.com>
According to the C99 specification section 7.20.3.2 paragraph 2:
> If ptr is a null pointer, no action occurs.
So we do not need to check that the pointer is a null pointer.
The following script leaks memory in Ripper:
```ruby
require "ripper"
20.times do
100_000.times do
Ripper.parse("")
end
puts `ps -o rss= -p #{$$}`
end
```
(https://github.com/ruby/fiddle/pull/123)
This commit adds two new methods, `Fiddle::Pointer.read` and
`Fiddle::Pointer.write`. Both methods take an address, and will read or
write bytes at that address respectively.
For example we can read from an address without making a Pointer object:
```ruby
Fiddle::Pointer.read(address, 5) # read 5 bytes
```
We can also write to an address without allocating a Pointer object:
```ruby
Fiddle::Pointer.write(address, "bytes") # write 5 bytes
```
This allows us to read / write memory at arbitrary addresses without
instantiating a new `Fiddle::Pointer` object.
Examples where this API would be useful
[1](f03481d28b/lib/tenderjit/fiddle_hacks.rb (L26-L28))
[2](77c8daa2d4/lib/ruby_vm/rjit/c_pointer.rb (L193))
[3](77c8daa2d4/lib/ruby_vm/rjit/c_pointer.rb (L284))
I also added a writer method for the same reasons as the reader.
---------
https://github.com/ruby/fiddle/commit/04238cefed
Co-authored-by: Sutou Kouhei <kou@clear-code.com>
clang generates a warning:
../../../../ext/openssl/ossl_pkey.c:326:22: error: implicit truncation from 'int' to a one-bit wide bit-field changes value from 1 to -1 [-Werror,-Wsingle-bit-bitfield-constant-conversion]
arg->interrupted = 1;
^ ~
1 error generated.
https://github.com/ruby/openssl/commit/4b2ba7b022
RUBY_OPENSSL_EXTCFLAGS and RUBY_OPENSSL_EXTLDFLAGS have been added for
the primary purpose of appending custom warning flags during
development and CI.
Since checking programs generated by mkmf may not be completely
warning-free, we don't want to apply -Werror that may be supplied from
those environment variables.
https://github.com/ruby/openssl/commit/2a95b971d5
This commit is a workaround to avoid the error below that the
`OpenSSL::PKey.read` fails with the OpenSSL 3.0 FIPS mode.
```
$ openssl genrsa -out key.pem 4096
$ ruby -e "require 'openssl'; OpenSSL::PKey.read(File.read('key.pem'))"
-e:1:in `read': Could not parse PKey (OpenSSL::PKey::PKeyError)
from -e:1:in `<main>'
```
The root cause is on the OpenSSL side. The `OSSL_DECODER_CTX_set_selection`
doesn't apply the selection value properly if there are multiple providers, and
a provider (e.g. "base" provider) handles the decoder implementation, and
another provider (e.g. "fips" provider) handles the keys.
The workaround is to create `OSSL_DECODER_CTX` variable each time without using
the `OSSL_DECODER_CTX_set_selection`.
https://github.com/ruby/openssl/commit/5ff4a31621
The vast majority have no reference so it's just a matter of setting the flags.
For the couple exception, they have very little references so it's
easy.
https://github.com/ruby/openssl/commit/2c7c6de69e
According to the `mkmf.rb#init_mkmf`, there are command line options below.
* `--with-cflags` to set the `cflags`
* `--with-ldflags` to set the `ldflags`
For example the following command compiles with the specified flags. Note that
`MAKEFLAGS` is to print the compiler command lines.
```
$ MAKEFLAGS="V=1" \
bundle exec rake compile -- \
--with-cflags="-Wundef -Werror" \
--with-ldflags="-fstack-protector"
```
However, I couldn't find command line options to append the flags. And this
commit is to append the `cflags` and `ldflags` by the environment variables.
```
$ MAKEFLAGS="V=1" \
RUBY_OPENSSL_EXTCFLAGS="-Wundef -Werror" \
RUBY_OPENSSL_EXTLDFLAGS="-fstack-protector" \
bundle exec rake compile
```
https://github.com/ruby/openssl/commit/b551eb86f6
Introduce Universal Parser mode for the parser.
This commit includes these changes:
* Introduce `UNIVERSAL_PARSER` macro. All of CRuby related functions
are passed via `struct rb_parser_config_struct` when this macro is enabled.
* Add CI task with 'cppflags=-DUNIVERSAL_PARSER' for ubuntu.
Sutou Kouhei
BurdetteLamar
Nobuyoshi Nakada
Jean Boussier
Hiroshi SHIBATA
Tim Kretschmer
Koichi ITO
Yusuke Endoh
Peter Zhu
Daisuke Aritomo
KJ Tsanaktsidis
KJ Tsanaktsidis
Samuel Williams
yui-knk
John Hawthorn
Jean byroot Boussier
Charles Oliver Nutter
Kazuki Yamaguchi
Michael Richardson
Petrik
Mau Magnaguagno
Jun Aruga
卜部昌平
Olivier Lacan
Sorah Fukumori
Benoit Daloze
Dimitar Haralanov
Ryo Kajiwara
cryptogopher
Aaron Patterson
qwyng