Граф коммитов

338 Коммитов

Автор SHA1 Сообщение Дата
Alan Wu f41b4d44f9 YJIT: Bounds check every byte in the assembler
Previously, YJIT assumed that basic blocks never consume more than
1 KiB of memory. This assumption does not hold for long Ruby methods
such as the one in the following:

```ruby
eval(<<RUBY)
def set_local_a_lot
  #{'_=0;'*0x40000}
end
RUBY

set_local_a_lot
```

For low `--yjit-exec-mem-size` values, one basic block could exhaust the
entire buffer.

Introduce a new field `codeblock_t::dropped_bytes` that the assembler
sets whenever it runs out of space. Check this field in
gen_single_block() to respond to out of memory situations and other
error conditions. This design avoids making the control flow graph of
existing code generation functions more complex.

Use POSIX shell in misc/test_yjit_asm.sh since bash is expanding
`0%/*/*` differently.

Co-authored-by: Aaron Patterson <tenderlove@ruby-lang.org>
2021-12-03 20:02:25 -05:00
Aaron Patterson 157095b3a4 Mark JIT code as writeable / executable depending on the situation
Some platforms don't want memory to be marked as writeable and
executable at the same time. When we write to the code block, we
calculate the OS page that the buffer position maps to.  Then we call
`mprotect` to allow writes on that particular page.  As an optimization,
we cache the "last written" aligned page which allows us to amortize the
cost of the `mprotect` call.  In other words, sequential writes to the
same page will only call `mprotect` on the page once.

When we're done writing, we call `mprotect` on the entire JIT buffer.
This means we don't need to keep track of which pages were marked as
writeable, we let the OS take care of that.

Co-authored-by: John Hawthorn <john@hawthorn.email>
2021-12-01 12:45:59 -08:00
Alan Wu 91a9062626
YJIT: use shorter encoding for mov(r64,imm) when unambiguous (#5081)
* YJIT: use shorter encoding for mov(r64,imm) when unambiguous

Previously, for small constants such as `mov(RAX, imm_opnd(Qundef))`,
we emit an instruction with an 8-byte immediate. This form commonly
gets the `movabs` mnemonic.

In 64-bit mode, 32-bit operands get zero extended to 64-bit to fill the
register, so when the immediate is small enough, we can save 4 bytes by
using the `mov` variant that takes a 32-bit immediate and does a zero
extension.

Not implement with this change, there is an imm32 variant of `mov` that
does sign extension we could use. When the constant is negative, we
fallback to the `movabs` form.

In railsbench, this change yields roughly a 12% code size reduction for
the outlined block.

Co-authored-by: Jemma Issroff <jemmaissroff@gmail.com>

* [ci skip] comment edit. Please squash.

Co-authored-by: Jemma Issroff <jemmaissroff@gmail.com>
2021-11-05 15:44:29 -04:00
Maxime Chevalier-Boisvert 2421527d6e
YJIT code pages refactoring for code GC (#5073)
* New code page allocation logic

* Fix leaked globals

* Fix leaked symbols, yjit asm tests

* Make COUNTED_EXIT take a jit argument, so we can eliminate global ocb

* Remove extra whitespace

* Change block start_pos/end_pos to be pointers instead of uint32_t

* Change branch end_pos and start_pos to end_addr, start_addr
2021-11-04 16:05:41 -04:00
Nobuyoshi Nakada a202408180
Fix typos 2021-11-02 19:17:37 +09:00
Nobuyoshi Nakada b74bf8dd88
Follow up the RString change [ci skip]
Since 46b66eb9e8, already `ary` has
been enclosed in `embed`.
2021-10-28 08:58:59 +09:00
Nobuyoshi Nakada 367884c659
Fix yjit_asm_tests.c as C99 compliant (#5033)
* rb_bug should be variadic

* Prefer ANSI-style prototypes over old K&R-style definitions

* Add missing argument types
2021-10-27 10:57:08 -04:00
Peter Zhu a5b6598192 [Feature #18239] Implement VWA for strings
This commit adds support for embedded strings with variable capacity and
uses Variable Width Allocation to allocate strings.
2021-10-25 13:26:23 -04:00
Nobuyoshi Nakada 4d4bdcf368
Move the test file 2021-10-21 13:07:48 +09:00
Nobuyoshi Nakada 58956dba16
Fix for out-of-place build 2021-10-21 13:07:48 +09:00
Alan Wu 00be5846e4 Fix non RUBY_DEBUG build warnings
On non RUBY_DEBUG builds, assert() compiles to nothing and the compiler
warns about uninitialized variables in those code paths. Replace
those asserts with rb_bug() to fix the warnings and do the assert in
all builds. Since yjit_asm_tests.c compiles outside of Ruby, it needed
a distinct version of rb_bug().

Also put YJIT_STATS check for function delcaration that is only defined
in YJIT_STATS builds.
2021-10-20 18:19:43 -04:00
Alan Wu 8a9a2d0049 Move test_yjit_asm.sh into misc
Since conventionally scripts don't live at the top level of the repo.
2021-10-20 18:19:43 -04:00
Aaron Patterson 09679f486c Add an lldb script to print YJIT comments
This script is an lldb helper that just loops through all the comments
stored and prints out the comment along with the address corresponding
to the comment.

For example, I'm crashing in JIT code at address 0x0000000110000168.
Using the `lc` helper I can see that it's probably crashing inside the
exit back to the interpreter

```
(lldb) bt 5
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x22220021)
    frame #0: 0x0000000110000168
  * frame #1: 0x00000001002b5ff5 miniruby`invoke_block_from_c_bh [inlined] invoke_block(ec=0x0000000100e05350, iseq=0x0000000100c1ff10, self=0x0000000100c76cc0, captured=<unavailable>, cref=0x0000000000000000, type=<unavailable>, opt_pc=<unavailable>) at vm.c:1268:12
    frame #2: 0x00000001002b5f7d miniruby`invoke_block_from_c_bh [inlined] invoke_iseq_block_from_c(ec=<unavailable>, captured=<unavailable>, self=0x0000000100c76cc0, argc=2, argv=<unavailable>, kw_splat=0, passed_block_handler=0x0000000000000000, cref=0x0000000000000000, is_lambda=<unavailable>, me=0x0000000000000000) at vm.c:1340
    frame #3: 0x00000001002b5e14 miniruby`invoke_block_from_c_bh(ec=<unavailable>, block_handler=<unavailable>, argc=<unavailable>, argv=<unavailable>, kw_splat=0, passed_block_handler=0x0000000000000000, cref=0x0000000000000000, is_lambda=<unavailable>, force_blockarg=0) at vm.c:1358
    frame #4: 0x000000010029860b miniruby`rb_yield_values(n=<unavailable>) at vm_eval.c:0
(lldb) lc
0x11000006d "putobject_INT2FIX_1_"
0x110000083 "leave"
0x110000087 "check for interrupts"
0x110000087 "RUBY_VM_CHECK_INTS(ec)"
0x110000098 "check for finish frame"
0x1100000ed "getlocal_WC_0"
0x110000107 "getlocal_WC_1"
0x11000012a "opt_send_without_block"
0x110000139 "opt_send_without_block"
0x11000013c "exit to interpreter"
```
2021-10-20 18:19:37 -04:00
Maxime Chevalier-Boisvert e7d20e6616 Free block->incoming in invalidate_block_version() 2021-10-20 18:19:29 -04:00
Maxime Chevalier-Boisvert d2ad0a1175 Added synthetic torture test with 30K tiny methods 2021-10-20 18:19:28 -04:00
Aaron Patterson e427fdff0a Directly link libcapstone for easier development
This lets us use libcapstone directly from miniruby so we don't need a
Ruby Gem to to dev work.

Example usage:

```ruby
def foo(x)
  if x < 1
    "wow"
  else
    "neat"
  end
end

iseq = RubyVM::InstructionSequence.of(method(:foo))
puts UJIT.disasm(iseq)
100.times { foo 1 }
puts UJIT.disasm(iseq)
```

Then in the terminal

```
$ ./miniruby test.rb

== disasm: #<ISeq:foo@test.rb:1 (1,0)-(7,3)> (catch: FALSE)
local table (size: 1, argc: 1 [opts: 0, rest: -1, post: 0, block: -1, kw: -1@-1, kwrest: -1])
[ 1] x@0<Arg>
0000 getlocal_WC_0                          x@0                       (   2)[LiCa]
0002 putobject_INT2FIX_1_
0003 opt_lt                                 <calldata!mid:<, argc:1, ARGS_SIMPLE>
0005 branchunless                           10
0007 putstring                              "wow"                     (   3)[Li]
0009 leave                                                            (   7)[Re]
0010 putstring                              "neat"                    (   5)[Li]
0012 leave                                                            (   7)[Re]

== ISEQ RANGE: 10 -> 10 ========================================================
        0x0:    movabs  rax, 0x7fe816e2d1a0
        0xa:    mov     qword ptr [rdi], rax
        0xd:    mov     r8, rax
        0x10:   mov     r9, rax
        0x13:   mov     r11, r12
        0x16:   jmp     qword ptr [rax]
== ISEQ RANGE: 0 -> 7 ==========================================================
        0x0:    mov     rax, qword ptr [rdi + 0x20]
        0x4:    mov     rax, qword ptr [rax - 0x18]
        0x8:    mov     qword ptr [rdx], rax
        0xb:    mov     qword ptr [rdx + 8], 3
        0x13:   movabs  rax, 0x7fe817808200
        0x1d:   test    byte ptr [rax + 0x3e6], 1
        0x24:   jne     0x3ffff7b
        0x2a:   test    byte ptr [rdx], 1
        0x2d:   je      0x3ffff7b
        0x33:   test    byte ptr [rdx + 8], 1
        0x37:   je      0x3ffff7b
        0x3d:   mov     rax, qword ptr [rdx]
        0x40:   cmp     rax, qword ptr [rdx + 8]
        0x44:   movabs  rax, 0
        0x4e:   movabs  rcx, 0x14
        0x58:   cmovl   rax, rcx
        0x5c:   mov     qword ptr [rdx], rax
        0x5f:   test    qword ptr [rdx], -9
        0x66:   jne     0x3ffffd5
```

Make sure to `brew install pkg-config capstone`
2021-10-20 18:19:27 -04:00
Maxime Chevalier-Boisvert d528cf4fd5 Added comments. Fixed compiler warning. 2021-10-20 18:19:27 -04:00
Aaron Patterson 11512a80fc add a helper script for ujit disasm 2021-10-20 18:19:27 -04:00
Nobuyoshi Nakada bbf98b572e
lldb: Get rid of error at unpreserved encodings [ci skip] 2021-09-29 22:31:24 +09:00
Nobuyoshi Nakada 545e01645f
lldb: Show encoding of String [ci skip] 2021-09-28 20:03:54 +09:00
Peter Zhu 62bc4a9420 [Feature #18045] Implement size classes for GC
This commits implements size classes in the GC for the Variable Width
Allocation feature. Unless `USE_RVARGC` compile flag is set, only a
single size class is created, maintaining current behaviour. See the
redmine ticket for more details.

Co-authored-by: Aaron Patterson <tenderlove@ruby-lang.org>
2021-08-25 09:28:21 -04:00
Peter Zhu c08d4067be [Feature #18045] Remove T_PAYLOAD
This commit removes T_PAYLOAD since the new VWA implementation no longer
requires T_PAYLOAD types.

Co-authored-by: Aaron Patterson <tenderlove@ruby-lang.org>
2021-08-25 09:28:21 -04:00
Peter Zhu eddd369e73 Revert "[Feature #18045] Implement size classes for GC"
This reverts commits 48ff7a9f3e
and b2e2cf2ded because it is causing
crashes in SPARC solaris and i386 debian.
2021-08-23 10:54:53 -04:00
Peter Zhu b2e2cf2ded [Feature #18045] Implement size classes for GC
This commits implements size classes in the GC for the Variable Width
Allocation feature. Unless `USE_RVARGC` compile flag is set, only a
single size class is created, maintaining current behaviour. See the
redmine ticket for more details.

Co-authored-by: Aaron Patterson <tenderlove@ruby-lang.org>
2021-08-23 09:15:42 -04:00
Peter Zhu 48ff7a9f3e [Feature #18045] Remove T_PAYLOAD
This commit removes T_PAYLOAD since the new VWA implementation no longer
requires T_PAYLOAD types.

Co-authored-by: Aaron Patterson <tenderlove@ruby-lang.org>
2021-08-23 09:15:42 -04:00
Nobuyoshi Nakada 91c542ad05
lldb_cruby.py: push non-flonum float to history [ci skip] 2021-06-04 09:24:57 +09:00
Nobuyoshi Nakada 3c57c087ec
lldb_cruby.py: fix non-flonum float inspection [ci skip] 2021-06-04 09:12:34 +09:00
Hiroshi SHIBATA be2e2b4805 Promote net-smtp to the bundled gems 2021-05-27 14:42:11 +09:00
Hiroshi SHIBATA aa9726f7b9 Promote net-pop to the bundled gems 2021-05-27 14:42:11 +09:00
Hiroshi SHIBATA d5bc6b2337 Promote net-imap to the bundled gems 2021-05-27 14:42:11 +09:00
Hiroshi SHIBATA e49c998d1e Promote net-ftp to the bundled gems 2021-05-27 14:42:11 +09:00
Hiroshi SHIBATA 454a36794f Promote matrix to the bundled gems 2021-05-27 14:42:11 +09:00
Hiroshi SHIBATA 43fb97dfba
Update the latest list of default gems for misc/expand_tabs.rb 2021-05-25 20:19:11 +09:00
Aaron Patterson 45ddefb14a
add rb_id2str to lldb debugging scripts 2021-05-24 16:02:42 -07:00
Peter Zhu 578e6416e7 lldb: convert heap_page_obj_limit from a float to int 2021-05-06 12:54:43 -04:00
Matt Valentine-House b0b7751f3b lldb: teach rp about T_PAYLOAD 2021-05-06 09:18:17 -04:00
Matt Valentine-House 5a451c4b1f lldb: Warn when attempting to dump invalid pages 2021-04-29 15:13:34 -04:00
Alexander Popov 2afbe7113a
[ruby/optparse] Add EditorConfig file
More info here: https://editorconfig.org/

For example, `ruby/ruby` has it: https://github.com/ruby/ruby/blob/05ebaee/.editorconfig

Also fix some offenses.

https://github.com/ruby/optparse/commit/29402e7e0e
2021-04-28 11:56:15 +09:00
Matt Valentine-House 1c1c91535c lldb: highlight the slot when using dump_page_rvalue 2021-04-27 10:58:49 -04:00
Matt Valentine-House f64bb9fc84 lldb: dump_page_rvalue - dump a heap page containing an RVALUE
rather than having to do this in a two step process:

1. heap_page obj
2. dump_page $2 (or whatever lldb variable heap_page set)

we can now just

dump_page_rvalue obj
2021-04-27 10:58:49 -04:00
Matt Valentine-House c752a35816 lldb: Add Freelist Index to dump_page output 2021-04-27 10:58:49 -04:00
Nobuyoshi Nakada 2bbae0e91a [ruby/optparse] Completion scripts themselves are not executable
https://github.com/ruby/optparse/commit/65d8aff935
2021-03-29 18:24:58 +09:00
Martin Rey d474b19b5b
[ruby/optparse] Use ZDOTDIR env var to locate .zshrc
https://github.com/ruby/optparse/commit/c4977674bf
2021-03-29 15:55:41 +09:00
Matt Valentine-House a47697aa44 LLDB: Introduce dump_page helper
This dumps out object type information for every object on a page in the
form:

bits [LM R ] T_CLASS    [389]: Addr: 0x1007ebcf0 (flags: 0x100000062)
2021-03-16 08:19:37 -07:00
Matt Valentine-House 1dca333599 LLDB: Extract a dump_bits function from rp
that dumps the heap page bitmaps for a slot
2021-03-16 08:19:37 -07:00
Hiroshi SHIBATA 0e73b49b4c Promote webrick to bundled gems 2020-12-10 18:06:25 +09:00
Aaron Patterson 8a06af5f88
Mostly recover a Ruby stack trace from a core file
Update the lldb script so it can mostly recover a Ruby stack trace from
a core file.  It's still missing line numbers and dealing with CFUNCs,
but you use it like this:

```
(lldb) rbbt ec
rb_control_frame_t TYPE
0x7f6fd6555fa0     EVAL   ./bootstraptest/runner.rb error!!
0x7f6fd6555f68     METHOD ./bootstraptest/runner.rb main
0x7f6fd6555f30     METHOD ./bootstraptest/runner.rb in_temporary_working_directory
0x7f6fd6555ef8     METHOD /home/aaron/git/ruby/lib/tmpdir.rb mktmpdir
0x7f6fd6555ec0     BLOCK  ./bootstraptest/runner.rb block in in_temporary_working_directory
0x7f6fd6555e88     CFUNC
0x7f6fd6555e50     BLOCK  ./bootstraptest/runner.rb block (2 levels) in in_temporary_working_directory
0x7f6fd6555e18     BLOCK  ./bootstraptest/runner.rb block in main
0x7f6fd6555de0     METHOD ./bootstraptest/runner.rb exec_test
0x7f6fd6555da8     CFUNC
0x7f6fd6555d70     BLOCK  ./bootstraptest/runner.rb block in exec_test
0x7f6fd6555d38     CFUNC
0x7f6fd6555d00     TOP    /home/aaron/git/ruby/bootstraptest/test_insns.rb error!!
0x7f6fd6555cc8     CFUNC
0x7f6fd6555c90     BLOCK  /home/aaron/git/ruby/bootstraptest/test_insns.rb block in <top (required)>
0x7f6fd6555c58     METHOD ./bootstraptest/runner.rb assert_equal
0x7f6fd6555c20     METHOD ./bootstraptest/runner.rb assert_check
0x7f6fd6555be8     METHOD ./bootstraptest/runner.rb show_progress
0x7f6fd6555bb0     METHOD ./bootstraptest/runner.rb with_stderr
0x7f6fd6555b78     BLOCK  ./bootstraptest/runner.rb block in show_progress
0x7f6fd6555b40     BLOCK  ./bootstraptest/runner.rb block in assert_check
0x7f6fd6555b08     METHOD ./bootstraptest/runner.rb get_result_string
0x7f6fd6555ad0     METHOD ./bootstraptest/runner.rb make_srcfile
0x7f6fd6555a98     CFUNC
0x7f6fd6555a60     BLOCK  ./bootstraptest/runner.rb block in make_srcfile
```

Getting the main execution context is difficult (it is stored in a
thread local) so for now you must supply an ec and this will make a
backtrace
2020-10-14 16:43:53 -07:00
Aaron Patterson bca8952fc7
Fix lldb disassembler so it works with core files
This fixes the lldb disassembler script so that it doesn't need a live
process when disassembling rb_iseq_t.  I also added the PC to the output
so you can tell what the VM is executing when it crashed.

For example:

```
(lldb) rbdisasm ec->cfp->iseq
PC             IDX  insn_name(operands)
0x56039f0a1720 0000 nop
0x56039f0a1728 0001 getlocal_WC_1( 5 )
0x56039f0a1738 0003 branchunless( 7 )
0x56039f0a1748 0005 getlocal_WC_0( 3 )
0x56039f0a1758 0007 putstring( (VALUE)0x56039f0c7eb8 )
0x56039f0a1768 0009 opt_send_without_block( (struct rb_call_data *)0x56039f09f140 )
0x56039f0a1778 0011 pop
0x56039f0a1780 0012 getglobal( ID: 0x7fd7 )
0x56039f0a1790 0014 branchunless( 7 )
0x56039f0a17a0 0016 getlocal_WC_0( 3 )
0x56039f0a17b0 0018 putstring( (VALUE)0x56039f0c7e90 )
0x56039f0a17c0 0020 opt_send_without_block( (struct rb_call_data *)0x56039f09f150 )
0x56039f0a17d0 0022 pop
0x56039f0a17d8 0023 getlocal_WC_0( 3 )
0x56039f0a17e8 0025 putobject( (VALUE)0x56039f0c7e68 )
0x56039f0a17f8 0027 getlocal_WC_1( 6 )
0x56039f0a1808 0029 dup
0x56039f0a1810 0030 checktype( 5 )
0x56039f0a1820 0032 branchif( 4 )
0x56039f0a1830 0034 dup
0x56039f0a1838 0035 opt_send_without_block( (struct rb_call_data *)0x56039f09f160 )
0x56039f0a1848 0037 tostring
0x56039f0a1850 0038 putobject( (VALUE)0x56039f0c7e40 )
0x56039f0a1860 0040 concatstrings( 3 )
0x56039f0a1870 0042 opt_send_without_block( (struct rb_call_data *)0x56039f09f170 )
0x56039f0a1880 0044 nop
0x56039f0a1888 0045 leave
(lldb) p ec->cfp->pc
(const VALUE *) $146 = 0x000056039f0a1848
```

Here we can see the VM is currently executing `opt_send_without_block`
(because the PC is one ahead of the current instruction)
2020-10-08 16:43:11 -07:00
Aaron Patterson 0a3099ae40
bit table information when printing an object 2020-09-28 16:45:19 -07:00
Aaron Patterson 3d474e19fd Rudimentary support for disassembling rb_iseq_t
I need to disassemble instruction sequences while debugging, so I wrote
this.

Usage is like this:

```
(lldb) p iseq
(rb_iseq_t *) $147 = 0x0000000101068400
(lldb) rbdisasm iseq
0000 putspecialobject( 3 )
0002 putnil
0003 defineclass( ID: 0x560b, (rb_iseq_t *)0x1010681d0, 2 )
0007 pop
0008 putspecialobject( 3 )
0010 putnil
0011 defineclass( ID: 0x56eb, (rb_iseq_t *)0x101063b58, 2 )
0015 leave
```

Also thanks a ton to @kivikakk helping me figure out how to navigate LLDB's Python 😆
2020-09-22 13:40:57 -07:00