ruby/benchmark

ruby/benchmark

This directory has benchmark definitions to be run with benchmark_driver.gem.

Normal usage

Execute gem install benchmark_driver and run a command like:

# Run a benchmark script with the ruby in the $PATH
benchmark-driver benchmark/erb_render.yml

# Run all benchmark scripts with multiple Ruby executables or options
benchmark-driver benchmark/*.yml -e /path/to/ruby -e '/path/to/ruby --jit'

# Or compare Ruby versions managed by rbenv
benchmark-driver benchmark/*.yml --rbenv '2.5.1;2.6.0-preview2 --jit'

# You can collect many metrics in many ways
benchmark-driver benchmark/*.yml --runner memory --output markdown

See also:

$ benchmark-driver --help
Usage: benchmark-driver [options] [YAML|RUBY]
    -r, --runner [TYPE]              Specify runner type: ips, time, memory, once (default: ips)
    -o, --output [TYPE]              Specify output type: compare, simple, markdown, record (default: compare)
    -e, --executables [EXECS]        Ruby executables (e1::path1,arg1,...; e2::path2,arg2;...)
        --rbenv [VERSIONS]           Ruby executables in rbenv (x.x.x,arg1,...;y.y.y,arg2,...;...)
        --repeat-count [NUM]         Try benchmark NUM times and use the fastest result or the worst memory usage
        --repeat-result [TYPE]       Yield "best", "average" or "worst" result with --repeat-count (default: best)
        --bundler                    Install and use gems specified in Gemfile
        --filter [REGEXP]            Filter out benchmarks with given regexp
        --run-duration [SECONDS]     Warmup estimates loop_count to run for this duration (default: 3)
    -v, --verbose                    Verbose mode. Multiple -v options increase visilibity (max: 2)

make benchmark

Using make benchmark, make update-benchmark-driver automatically downloads the supported version of benchmark_driver, and it runs benchmarks with the downloaded benchmark-driver.

# Run all benchmarks with the ruby in the $PATH and the built ruby
make benchmark

# Or compare with specific ruby binary
make benchmark COMPARE_RUBY="/path/to/ruby --jit"

# Run vm1 benchmarks
make benchmark ITEM=vm1

# Run some limited benchmarks in ITEM-matched files
make benchmark ITEM=vm1 OPTS=--filter=block

# You can specify the benchmark by an exact filename instead of using the default argument:
# ARGS = $$(find $(srcdir)/benchmark -maxdepth 1 -name '*$(ITEM)*.yml' -o -name '*$(ITEM)*.rb')
make benchmark ARGS=../benchmark/erb_render.yml

# You can specify any option via $OPTS
make benchmark OPTS="--help"

# With `make benchmark`, some special runner plugins are available:
#   -r peak, -r size, -r total, -r utime, -r stime, -r cutime, -r cstime
make benchmark ITEM=vm2_bigarray OPTS="-r peak"