Jean Boussier
|
6e2619c968
|
Revamp the benchmark suite
There is a large number of outstanding performance PRs that I want to
merge, but we need a decent benchmark to judge if they are effective.
I went to borrow rapidjson's benchmark suite, which is a good start.
I only kept the comparison with Oj and RapidJSON, because YAJL is
slower on most benchmarks, so little point comparing to it.
Encoding:
```
== Encoding small nested array (121 bytes)
ruby 3.3.4 (2024-07-09 revision be1089c8ec) +YJIT [arm64-darwin23]
Warming up --------------------------------------
json 88.225k i/100ms
oj 209.862k i/100ms
rapidjson 128.978k i/100ms
Calculating -------------------------------------
json 914.611k (± 0.4%) i/s (1.09 μs/i) - 4.588M in 5.016099s
oj 2.163M (± 0.2%) i/s (462.39 ns/i) - 10.913M in 5.045964s
rapidjson 1.392M (± 1.3%) i/s (718.55 ns/i) - 6.965M in 5.005438s
Comparison:
json: 914610.6 i/s
oj: 2162693.5 i/s - 2.36x faster
rapidjson: 1391682.6 i/s - 1.52x faster
== Encoding small hash (65 bytes)
ruby 3.3.4 (2024-07-09 revision be1089c8ec) +YJIT [arm64-darwin23]
Warming up --------------------------------------
json 142.093k i/100ms
oj 651.412k i/100ms
rapidjson 237.706k i/100ms
Calculating -------------------------------------
json 1.478M (± 0.7%) i/s (676.78 ns/i) - 7.389M in 5.000866s
oj 7.150M (± 0.7%) i/s (139.85 ns/i) - 35.828M in 5.010756s
rapidjson 2.250M (± 1.6%) i/s (444.46 ns/i) - 11.410M in 5.072451s
Comparison:
json: 1477595.1 i/s
oj: 7150472.0 i/s - 4.84x faster
rapidjson: 2249926.7 i/s - 1.52x faster
== Encoding twitter.json (466906 bytes)
ruby 3.3.4 (2024-07-09 revision be1089c8ec) +YJIT [arm64-darwin23]
Warming up --------------------------------------
json 101.000 i/100ms
oj 223.000 i/100ms
rapidjson 105.000 i/100ms
Calculating -------------------------------------
json 1.017k (± 0.7%) i/s (982.83 μs/i) - 5.151k in 5.062786s
oj 2.244k (± 0.7%) i/s (445.72 μs/i) - 11.373k in 5.069428s
rapidjson 1.069k (± 4.6%) i/s (935.20 μs/i) - 5.355k in 5.016652s
Comparison:
json: 1017.5 i/s
oj: 2243.6 i/s - 2.21x faster
rapidjson: 1069.3 i/s - same-ish: difference falls within error
== Encoding citm_catalog.json (500299 bytes)
ruby 3.3.4 (2024-07-09 revision be1089c8ec) +YJIT [arm64-darwin23]
Warming up --------------------------------------
json 77.000 i/100ms
oj 129.000 i/100ms
rapidjson 96.000 i/100ms
Calculating -------------------------------------
json 767.217 (± 2.5%) i/s (1.30 ms/i) - 3.850k in 5.021957s
oj 1.291k (± 1.5%) i/s (774.45 μs/i) - 6.579k in 5.096439s
rapidjson 959.527 (± 1.1%) i/s (1.04 ms/i) - 4.800k in 5.003052s
Comparison:
json: 767.2 i/s
oj: 1291.2 i/s - 1.68x faster
rapidjson: 959.5 i/s - 1.25x faster
== Encoding canada.json (2090234 bytes)
ruby 3.3.4 (2024-07-09 revision be1089c8ec) +YJIT [arm64-darwin23]
Warming up --------------------------------------
json 1.000 i/100ms
oj 3.000 i/100ms
rapidjson 1.000 i/100ms
Calculating -------------------------------------
json 19.748 (± 0.0%) i/s (50.64 ms/i) - 99.000 in 5.013336s
oj 31.016 (± 0.0%) i/s (32.24 ms/i) - 156.000 in 5.029732s
rapidjson 19.419 (± 0.0%) i/s (51.50 ms/i) - 98.000 in 5.050382s
Comparison:
json: 19.7 i/s
oj: 31.0 i/s - 1.57x faster
rapidjson: 19.4 i/s - 1.02x slower
== Encoding many #to_json calls (2661 bytes)
oj does not match expected output. Skipping
rapidjson unsupported (Invalid object key type: Object)
ruby 3.3.4 (2024-07-09 revision be1089c8ec) +YJIT [arm64-darwin23]
Warming up --------------------------------------
json 2.129k i/100ms
Calculating -------------------------------------
json 21.599k (± 0.6%) i/s (46.30 μs/i) - 108.579k in 5.027198s
```
Parsing:
```
== Parsing small nested array (121 bytes)
ruby 3.3.4 (2024-07-09 revision be1089c8ec) +YJIT [arm64-darwin23]
Warming up --------------------------------------
json 47.497k i/100ms
oj 54.115k i/100ms
oj strict 53.854k i/100ms
Oj::Parser 150.904k i/100ms
rapidjson 80.775k i/100ms
Calculating -------------------------------------
json 481.096k (± 1.1%) i/s (2.08 μs/i) - 2.422M in 5.035657s
oj 554.878k (± 0.6%) i/s (1.80 μs/i) - 2.814M in 5.071521s
oj strict 547.888k (± 0.7%) i/s (1.83 μs/i) - 2.747M in 5.013212s
Oj::Parser 1.545M (± 0.4%) i/s (647.16 ns/i) - 7.847M in 5.078302s
rapidjson 822.422k (± 0.6%) i/s (1.22 μs/i) - 4.120M in 5.009178s
Comparison:
json: 481096.4 i/s
Oj::Parser: 1545223.5 i/s - 3.21x faster
rapidjson: 822422.4 i/s - 1.71x faster
oj: 554877.7 i/s - 1.15x faster
oj strict: 547887.7 i/s - 1.14x faster
== Parsing small hash (65 bytes)
ruby 3.3.4 (2024-07-09 revision be1089c8ec) +YJIT [arm64-darwin23]
Warming up --------------------------------------
json 154.479k i/100ms
oj 220.283k i/100ms
oj strict 249.928k i/100ms
Oj::Parser 445.062k i/100ms
rapidjson 289.615k i/100ms
Calculating -------------------------------------
json 1.581M (± 3.0%) i/s (632.55 ns/i) - 8.033M in 5.086476s
oj 2.202M (± 3.5%) i/s (454.08 ns/i) - 11.014M in 5.008146s
oj strict 2.498M (± 3.5%) i/s (400.25 ns/i) - 12.496M in 5.008245s
Oj::Parser 4.640M (± 0.4%) i/s (215.50 ns/i) - 23.588M in 5.083443s
rapidjson 3.111M (± 0.3%) i/s (321.44 ns/i) - 15.639M in 5.027097s
Comparison:
json: 1580898.5 i/s
Oj::Parser: 4640298.1 i/s - 2.94x faster
rapidjson: 3111005.2 i/s - 1.97x faster
oj strict: 2498421.4 i/s - 1.58x faster
oj: 2202276.6 i/s - 1.39x faster
== Parsing test from oj (256 bytes)
ruby 3.3.4 (2024-07-09 revision be1089c8ec) +YJIT [arm64-darwin23]
Warming up --------------------------------------
json 37.580k i/100ms
oj 41.899k i/100ms
oj strict 50.731k i/100ms
Oj::Parser 74.589k i/100ms
rapidjson 50.954k i/100ms
Calculating -------------------------------------
json 382.150k (± 1.0%) i/s (2.62 μs/i) - 1.917M in 5.015737s
oj 420.282k (± 0.2%) i/s (2.38 μs/i) - 2.137M in 5.084338s
oj strict 511.758k (± 0.5%) i/s (1.95 μs/i) - 2.587M in 5.055821s
Oj::Parser 759.087k (± 0.3%) i/s (1.32 μs/i) - 3.804M in 5.011388s
rapidjson 518.273k (± 1.8%) i/s (1.93 μs/i) - 2.599M in 5.015867s
Comparison:
json: 382149.6 i/s
Oj::Parser: 759087.1 i/s - 1.99x faster
rapidjson: 518272.8 i/s - 1.36x faster
oj strict: 511758.4 i/s - 1.34x faster
oj: 420282.5 i/s - 1.10x faster
== Parsing twitter.json (567916 bytes)
ruby 3.3.4 (2024-07-09 revision be1089c8ec) +YJIT [arm64-darwin23]
Warming up --------------------------------------
json 52.000 i/100ms
oj 63.000 i/100ms
oj strict 74.000 i/100ms
Oj::Parser 79.000 i/100ms
rapidjson 56.000 i/100ms
Calculating -------------------------------------
json 522.896 (± 0.4%) i/s (1.91 ms/i) - 2.652k in 5.071809s
oj 624.849 (± 0.6%) i/s (1.60 ms/i) - 3.150k in 5.041398s
oj strict 737.779 (± 0.4%) i/s (1.36 ms/i) - 3.700k in 5.015117s
Oj::Parser 789.254 (± 0.3%) i/s (1.27 ms/i) - 3.950k in 5.004764s
rapidjson 565.663 (± 0.4%) i/s (1.77 ms/i) - 2.856k in 5.049015s
Comparison:
json: 522.9 i/s
Oj::Parser: 789.3 i/s - 1.51x faster
oj strict: 737.8 i/s - 1.41x faster
oj: 624.8 i/s - 1.19x faster
rapidjson: 565.7 i/s - 1.08x faster
== Parsing citm_catalog.json (1727030 bytes)
ruby 3.3.4 (2024-07-09 revision be1089c8ec) +YJIT [arm64-darwin23]
Warming up --------------------------------------
json 27.000 i/100ms
oj 31.000 i/100ms
oj strict 36.000 i/100ms
Oj::Parser 42.000 i/100ms
rapidjson 38.000 i/100ms
Calculating -------------------------------------
json 305.248 (± 0.3%) i/s (3.28 ms/i) - 1.539k in 5.041813s
oj 320.265 (± 3.4%) i/s (3.12 ms/i) - 1.612k in 5.039715s
oj strict 373.701 (± 1.6%) i/s (2.68 ms/i) - 1.872k in 5.010633s
Oj::Parser 457.792 (± 0.4%) i/s (2.18 ms/i) - 2.310k in 5.046049s
rapidjson 350.933 (± 8.8%) i/s (2.85 ms/i) - 1.748k in 5.052491s
Comparison:
json: 305.2 i/s
Oj::Parser: 457.8 i/s - 1.50x faster
oj strict: 373.7 i/s - 1.22x faster
rapidjson: 350.9 i/s - 1.15x faster
oj: 320.3 i/s - 1.05x faster
== Parsing canada.json (2251051 bytes)
ruby 3.3.4 (2024-07-09 revision be1089c8ec) +YJIT [arm64-darwin23]
Warming up --------------------------------------
json 2.000 i/100ms
oj 2.000 i/100ms
oj strict 2.000 i/100ms
Oj::Parser 2.000 i/100ms
rapidjson 28.000 i/100ms
Calculating -------------------------------------
json 29.216 (± 6.8%) i/s (34.23 ms/i) - 146.000 in 5.053753s
oj 24.899 (± 0.0%) i/s (40.16 ms/i) - 126.000 in 5.061915s
oj strict 24.828 (± 4.0%) i/s (40.28 ms/i) - 124.000 in 5.003067s
Oj::Parser 30.867 (± 3.2%) i/s (32.40 ms/i) - 156.000 in 5.057104s
rapidjson 285.761 (± 1.0%) i/s (3.50 ms/i) - 1.456k in 5.095715s
Comparison:
json: 29.2 i/s
rapidjson: 285.8 i/s - 9.78x faster
Oj::Parser: 30.9 i/s - same-ish: difference falls within error
oj: 24.9 i/s - 1.17x slower
oj strict: 24.8 i/s - 1.18x slower
```
|
2024-10-08 14:18:37 +00:00 |