LibreSSL 3.4 added EVP_DigestSign() and EVP_DigestVerify(). Use them
when available to prepare for the addition of Ed25519 support in
LibreSSL 3.7.
https://github.com/ruby/openssl/commit/475b2bf766
I see several arguments in doing so.
First they use a non trivial amount of memory, so for various memory
profiling/mapping tools it is relevant to have visibility of the space
occupied by shapes.
Then, some pathological code can create a tons of shape, so it is
valuable to have a way to have a way to observe shapes without having
to compile Ruby with `SHAPE_DEBUG=1`.
And additionally it's likely much faster to dump then this way than
to use `RubyVM::Shape`.
There are however a few open questions:
- Shapes can't respect the `since:` argument. Not sure what to do when
it is provided. Would probably make sense to not dump them.
- Maybe it would make more sense to have a separate `ObjectSpace.dump_shapes`?
- Maybe instead `dump_all` should take a `shapes: false` argument?
Additionally, `ObjectSpace.dump_shapes` is added for the use case of
debugging the evolution of the shape tree.
Implementation for Language Server Protocol (LSP) sometimes needs token information.
For example both `m(1)` and `m(1, )` has same AST structure other than node locations
then it's impossible to check the existence of `,` from AST. However in later case,
it might be better to suggest variables list for the second argument.
Token information is important for such case.
This commit adds these methods.
* Add `keep_tokens` option for `RubyVM::AbstractSyntaxTree.parse`, `.parse_file` and `.of`
* Add `RubyVM::AbstractSyntaxTree::Node#tokens` which returns tokens for the node including tokens for descendants nodes.
* Add `RubyVM::AbstractSyntaxTree::Node#all_tokens` which returns all tokens for the input script regardless the receiver node.
[Feature #19070]
Impacts on memory usage and performance are below:
Memory usage:
```
$ cat test.rb
root = RubyVM::AbstractSyntaxTree.parse_file(File.expand_path('../test/ruby/test_keyword.rb', __FILE__), keep_tokens: true)
$ /usr/bin/time -f %Mkb /usr/local/bin/ruby -v
ruby 3.2.0dev (2022-11-19T09:41:54Z 19070-keep_tokens d3af1b8057) [x86_64-linux]
11408kb
# keep_tokens :false
$ /usr/bin/time -f %Mkb /usr/local/bin/ruby test.rb
17508kb
# keep_tokens :true
$ /usr/bin/time -f %Mkb /usr/local/bin/ruby test.rb
30960kb
```
Performance:
```
$ cat ../ast_keep_tokens.yml
prelude: |
src = <<~SRC
module M
class C
def m1(a, b)
1 + a + b
end
end
end
SRC
benchmark:
without_keep_tokens: |
RubyVM::AbstractSyntaxTree.parse(src, keep_tokens: false)
with_keep_tokens: |
RubyVM::AbstractSyntaxTree.parse(src, keep_tokens: true)
$ make benchmark COMPARE_RUBY="./ruby" ARGS=../ast_keep_tokens.yml
/home/kaneko.y/.rbenv/shims/ruby --disable=gems -rrubygems -I../benchmark/lib ../benchmark/benchmark-driver/exe/benchmark-driver \
--executables="compare-ruby::./ruby -I.ext/common --disable-gem" \
--executables="built-ruby::./miniruby -I../lib -I. -I.ext/common ../tool/runruby.rb --extout=.ext -- --disable-gems --disable-gem" \
--output=markdown --output-compare -v ../ast_keep_tokens.yml
compare-ruby: ruby 3.2.0dev (2022-11-19T09:41:54Z 19070-keep_tokens d3af1b8057) [x86_64-linux]
built-ruby: ruby 3.2.0dev (2022-11-19T09:41:54Z 19070-keep_tokens d3af1b8057) [x86_64-linux]
warming up..
| |compare-ruby|built-ruby|
|:--------------------|-----------:|---------:|
|without_keep_tokens | 21.659k| 21.303k|
| | 1.02x| -|
|with_keep_tokens | 6.220k| 5.691k|
| | 1.09x| -|
```
* Windows: Fix warning about undefined if_indextoname()
* Windows: Fix UNIXSocket on MINGW and make .pair more reliable
* Windows: Use nonblock=true for read tests with scheduler
* Windows: Move socket detection from File.socket? to File.stat
Add S_IFSOCK to Windows and interpret reparse points accordingly.
Enable tests that work now.
* Windows: Use wide-char functions to UNIXSocket
This fixes behaviour with non-ASCII characters.
It also fixes deletion of temporary UNIXSocket.pair files.
* Windows: Add UNIXSocket tests for specifics of Windows impl.
* Windows: fix VC build due to missing _snwprintf
Avoid usage of _snwprintf, since it fails linking ruby.dll like so:
linking shared-library x64-vcruntime140-ruby320.dll
x64-vcruntime140-ruby320.def : error LNK2001: unresolved external symbol snwprintf
x64-vcruntime140-ruby320.def : error LNK2001: unresolved external symbol vsnwprintf_l
whereas linking miniruby.exe succeeds.
This patch uses snprintf on the UTF-8 string instead.
Also remove branch GetWindowsDirectoryW, since it doesn't work.
* Windows: Fix dangling symlink test failures
Co-authored-by: Lars Kanis <kanis@comcard.de>
* Add rbd_allocate_struct_zero for making 0.0
* Add rbd_allocate_struct_one for making 1.0
* Use them to replace VpAlloc calls
* Renmae VpPt5 to VpConstPt5
https://github.com/ruby/bigdecimal/commit/40c826f5e6
Since object shapes store the capacity of an object, we no longer
need the numiv field on RObjects. This gives us one extra slot which
we can use to give embedded objects one more instance variable (for a
total of 3 ivs). This commit removes the concept of numiv from RObject.
This commit adds a `capacity` field to shapes, and adds shape
transitions whenever an object's capacity changes. Objects which are
allocated out of a bigger size pool will also make a transition from the
root shape to the shape with the correct capacity for their size pool
when they are allocated.
This commit will allow us to remove numiv from objects completely, and
will also mean we can guarantee that if two objects share shapes, their
IVs are in the same positions (an embedded and extended object cannot
share shapes). This will enable us to implement ivar sets in YJIT using
object shapes.
Co-Authored-By: Aaron Patterson <tenderlove@ruby-lang.org>
Fixes the following issue when compiling using C99:
ext/-test-/rb_call_super_kw/rb_call_super_kw.c
ext/-test-/random/loop.c:16:39: error: extra ';' outside of a function [-Werror,-Wextra-semi]
RB_RANDOM_DEFINE_INIT_INT32_FUNC(loop);
because it's much slower on M1 https://github.com/ruby/erb/pull/29.
It'd be too complicated to switch the implementation based on known
optimized platforms / versions.
Besides, short strings are the most common usages of this method and
SIMD doesn't really help that case. All in all, I can't justify the
existence of this code.
https://github.com/ruby/erb/commit/30691c8995
(https://github.com/ruby/erb/pull/29)
Typically, strpbrk(3) is optimized pretty well with SIMD instructions.
Just using it makes this as fast as a SIMD-based implementation for the
no-escape case.
Not utilizing this for escaped cases because memory allocation would be
a more significant bottleneck for many strings anyway. Also, there'll be
some overhead in calling a C function (strpbrk) many times because we're
not using SIMD instructions directly. So using strpbrk all the time
might not necessarily be faster.
So different timestamps for different paths will be used. Extentions
paths in bundled gems contain `ruby_version`, which includes the ABI
version, and the same timestamp file for different paths resulted in
build failures when it changed.
- This callback is invoked when TLS key material is generated or
received, in order to allow applications to store this keying material
for debugging purposes.
- It is invoked with an `SSLSocket` and a string containing the key
material in the format used by NSS for its SSLKEYLOGFILE debugging
output.
- This commit adds the Ruby binding `keylog_cb` and the related tests
- It is only compatible with OpenSSL >= 1.1.1. Even if LibreSSL implements
`SSL_CTX_set_keylog_callback()` from v3.4.2, it does nothing (see
648d39f0f0)
https://github.com/ruby/openssl/commit/3b63232cf1
Scan through the input for a private key, then fallback to generic
decoder.
OpenSSL 3.0's OSSL_DECODER supports encoded key parameters. The PEM
header "-----BEGIN EC PARAMETERS-----" is used by one of such encoding
formats. While this is useful for OpenSSL::PKey::PKey, an edge case has
been discovered.
The openssl CLI command line "openssl ecparam -genkey" prints two PEM
blocks in a row, one for EC parameters and another for the private key.
Feeding the whole output into OSSL_DECODER results in only the first PEM
block, the key parameters, being decoded. Previously, ruby/openssl did
not support decoding key parameters and it would decode the private key
PEM block instead.
While the new behavior is technically correct, "openssl ecparam -genkey"
is so widely used that ruby/openssl does not want to break existing
applications.
Fixes https://github.com/ruby/openssl/pull/535https://github.com/ruby/openssl/commit/d486c82833
Current OpenSSL 3.0.x release has a regression with zero-length MAC
keys. While this issue should be fixed in a future release of OpenSSL,
we can use EVP_PKEY_new_raw_private_key() in place of the problematic
EVP_PKEY_new_mac_key() to avoid the issue. OpenSSL 3.0's man page
recommends using it regardless:
> EVP_PKEY_new_mac_key() works in the same way as
> EVP_PKEY_new_raw_private_key(). New applications should use
> EVP_PKEY_new_raw_private_key() instead.
Fixes https://github.com/ruby/openssl/issues/369#issuecomment-1224912710https://github.com/ruby/openssl/commit/4293f18b1f
X509at_delete_attr() in OpenSSL master puts an error queue entry if
there is no attribute left to delete. We must either clear the error
queue, or try not to call it when the list is already empty.
https://github.com/ruby/openssl/commit/a0c878481f
X509_STORE_get_ex_new_index() is a macro, so passing just its name to
have_func() doesn't detect it. Pass an example call instead.
https://github.com/ruby/openssl/commit/8d264d3e60
Co-authored-by: Nobuyoshi Nakada <nobu@ruby-lang.org>
While building with a custom build of OpenSSL, I noticed in mkmf.log
that all the feature detection checks are done using a program lacking
an OpenSSL header include. `mkmf` retries using a fallback program when
this fails, but that means all the `have_func` calls compile twice when
compiling once should suffice. Example log without this commit:
have_func: checking for X509_STORE_CTX_get0_cert()... -------------------- yes
DYLD_FALLBACK_LIBRARY_PATH=.:../.. "clang -o conftest ...
conftest.c:14:57: error: use of undeclared identifier 'X509_STORE_CTX_get0_cert'
int t(void) { void ((*volatile p)()); p = (void ((*)()))X509_STORE_CTX_get0_cert; return !p; }
^
1 error generated.
checked program was:
/* begin */
1: #include "ruby.h"
2:
3: /*top*/
4: extern int t(void);
5: int main(int argc, char **argv)
6: {
7: if (argc > 1000000) {
8: int (* volatile tp)(void)=(int (*)(void))&t;
9: printf("%d", (*tp)());
10: }
11:
12: return !!argv[argc];
13: }
14: int t(void) { void ((*volatile p)()); p = (void ((*)()))X509_STORE_CTX_get0_cert; return !p; }
/* end */
DYLD_FALLBACK_LIBRARY_PATH=.:../.. "clang -o conftest ...
checked program was:
/* begin */
1: #include "ruby.h"
2:
3: /*top*/
4: extern int t(void);
5: int main(int argc, char **argv)
6: {
7: if (argc > 1000000) {
8: int (* volatile tp)(void)=(int (*)(void))&t;
9: printf("%d", (*tp)());
10: }
11:
12: return !!argv[argc];
13: }
14: extern void X509_STORE_CTX_get0_cert();
15: int t(void) { X509_STORE_CTX_get0_cert(); return 0; }
/* end */
The second compilation succeeds.
Specify the header for each checked function.
https://github.com/ruby/openssl/commit/34ae7d92d0
openssl has to support older versions of Ruby. Undo the change in
ext/openssl/ossl_pkey_ec.c by commit efb91ff19b ("Rename
rb_ary_tmp_new to rb_ary_hidden_new", 2022-07-25).
By this change, syntax error is recovered smaller units.
In the case below, "DEFN :bar" is same level with "CLASS :Foo"
now.
```
module Z
class Foo
foo.
end
def bar
end
end
```
[Feature #19013]
I would like to check if a symbol is defined before trying to access it.
Some symbols aren't available on all platforms, so instead of raising an
exception, I want to check if it's defined first.
Today we have to do:
```ruby
begin
addr = Fiddle::Handle.sym("something")
# do something
rescue Fiddle::DLError
end
```
I want to write this:
```ruby
if Fiddle::Handle.sym_defined?("something")
addr = Fiddle::Handle.sym("something")
# do something
end
```
https://github.com/ruby/fiddle/commit/9d3371de13
Co-authored-by: Sutou Kouhei <kou@clear-code.com>
This helps to reduce repetition in code. Instead of doing "TYPE_*"
everywhere, you can do `include Fiddle::Types`, and write the type name
directly.
This PR is to help reduce repetition when writing Fiddle code. Right now
we have to type `TYPE_` everywhere, and you also have to include all of
`Fiddle` to access `TYPE_*` constants. With this change, you can just
include `Fiddle::Types` and it will shorten your code and also you only
have to include those constants.
Here is an example before:
```ruby
require "fiddle"
module MMAP
# All Fiddle constants included
include Fiddle
def self.make_function name, args, ret
ptr = Handle::DEFAULT[name]
func = Function.new ptr, args, ret, name: name
define_singleton_method name, &func.to_proc
end
make_function "munmap", [TYPE_VOIDP, # addr
TYPE_SIZE_T], # len
TYPE_INT
make_function "mmap", [TYPE_VOIDP,
TYPE_SIZE_T,
TYPE_INT,
TYPE_INT,
TYPE_INT,
TYPE_INT], TYPE_VOIDP
make_function "mprotect", [TYPE_VOIDP, TYPE_SIZE_T, TYPE_INT], TYPE_INT
end
```
After:
```ruby
require "fiddle"
module MMAP
# Only type names included
include Fiddle::Types
def self.make_function name, args, ret
ptr = Fiddle::Handle::DEFAULT[name]
func = Fiddle::Function.new ptr, args, ret, name: name
define_singleton_method name, &func.to_proc
end
make_function "munmap", [VOIDP, # addr
SIZE_T], # len
INT
make_function "mmap", [VOIDP, SIZE_T, INT, INT, INT, INT], VOIDP
make_function "mprotect", [VOIDP, SIZE_T, INT], INT
end
```
We only need to import the type names, and you don't have to type
`TYPE_` over and over. I think this makes Fiddle code easier to read.
https://github.com/ruby/fiddle/commit/49fa7233e5
Co-authored-by: Sutou Kouhei <kou@clear-code.com>
This commit adds constants for unsigned values. Currently we can use `-`
to mean "unsigned", but I think having a specific name makes Fiddle more
user friendly. This commit continues to support `-`, but introduces
negative constants with "unsigned" names
I think this will help to eliminate [this
code](3a56bf0bcc/lib/mjit/c_type.rb (L31-L38))
https://github.com/ruby/fiddle/commit/2bef0f1082
Co-authored-by: Sutou Kouhei <kou@clear-code.com>
Moves Expect library doc into io.c.
Changes certain links to local sections, now pointing to sections in doc/io_streams.rdoc.
Removes local sections now superseded by sections in doc/io_streams.rdoc.
Object Shapes is used for accessing instance variables and representing the
"frozenness" of objects. Object instances have a "shape" and the shape
represents some attributes of the object (currently which instance variables are
set and the "frozenness"). Shapes form a tree data structure, and when a new
instance variable is set on an object, that object "transitions" to a new shape
in the shape tree. Each shape has an ID that is used for caching. The shape
structure is independent of class, so objects of different types can have the
same shape.
For example:
```ruby
class Foo
def initialize
# Starts with shape id 0
@a = 1 # transitions to shape id 1
@b = 1 # transitions to shape id 2
end
end
class Bar
def initialize
# Starts with shape id 0
@a = 1 # transitions to shape id 1
@b = 1 # transitions to shape id 2
end
end
foo = Foo.new # `foo` has shape id 2
bar = Bar.new # `bar` has shape id 2
```
Both `foo` and `bar` instances have the same shape because they both set
instance variables of the same name in the same order.
This technique can help to improve inline cache hits as well as generate more
efficient machine code in JIT compilers.
This commit also adds some methods for debugging shapes on objects. See
`RubyVM::Shape` for more details.
For more context on Object Shapes, see [Feature: #18776]
Co-Authored-By: Aaron Patterson <tenderlove@ruby-lang.org>
Co-Authored-By: Eileen M. Uchitelle <eileencodes@gmail.com>
Co-Authored-By: John Hawthorn <john@hawthorn.email>
zverok
Nobuyoshi Nakada
Hiroshi SHIBATA
Henrique Bontempo
Theo Buehler
Yusuke Nakamura
Peter Zhu
Jean Boussier
Charles Oliver Nutter
Sutou Kouhei
Daniel Colson
Jemma Issroff
S-H-GAMELINKS
Takashi Kokubun
Yudai Takada
yui-knk
Samuel Williams
Kenta Murata
Burdette Lamar
Aaron Patterson
Ben Toews
Bart de Water
Christophe De La Fuente
Kazuki Yamaguchi
madblobfish
Nobuhiro IMAI
Alan Wu