# benchmarks
**Repository Path**: iqdo/benchmarks
## Basic Information
- **Project Name**: benchmarks
- **Description**: No description available
- **Primary Language**: Unknown
- **License**: MIT
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No
## Statistics
- **Stars**: 0
- **Forks**: 0
- **Created**: 2024-07-05
- **Last Updated**: 2024-07-05
## Categories & Tags
**Categories**: Uncategorized
**Tags**: None
## README
# Table of Content
* [Overview](#overview)
* [Measurements](#measurements)
* [Test Cases](#test-cases)
* [Brainfuck](#brainfuck)
* [bench.b](#benchb)
* [mandel.b](#mandelb)
* [Base64](#base64)
* [Json](#json)
* [Matmul](#matmul)
* [Primes](#primes)
* [Tests Execution](#tests-execution)
* [Environment](#environment)
* [Using Docker](#using-docker)
* [Manual Execution](#manual-execution)
* [Prerequisites](#prerequisites)
* [Contribution](#contribution)
* [Makefile guide](#makefile-guide)
* [Binary executables](#binary-executables)
* [Compiled artifacts](#compiled-artifacts)
* [Scripting language](#scripting-language)
* [README update](#readme-update)
* [Docker image update](#docker-image-update)
# Overview
The benchmarks follow the criteria:
- They are written as the average software developer would write them, i.e.
- The algorithms are implemented as cited in public sources;
- The libraries are used as described in the tutorials, documentation and examples;
- The used data structures are idiomatic.
- The used algorithms are similar between the languages (as the reference implementations), variants are acceptable if the reference implementation exists.
- All final binaries are releases (optimized for performance if possible) as debug performance may vary too much depending on the compiler.
My other benchmarks: [jit-benchmarks](https://github.com/kostya/jit-benchmarks), [crystal-benchmarks-game](https://github.com/kostya/crystal-benchmarks-game)
## Measurements
The measured values are:
- time spent for the benchmark execution (loading required data and code self-testing are not measured);
- memory consumption of the benchmark process, reported as `base` + `increase`, where `base` is the RSS before the benchmark and `increase` is the peak increase of the RSS during the benchmark;
- energy consumption of the CPU package during the benchmark: PP0 (cores) + PP1 (uncores like GPU) + DRAM. Currently, only Intel CPU are supported via the powercap interface.
All values are presented as: `median`±`median absolute deviation`.
UPDATE: 2024-06-06
# Test Cases
## Brainfuck
Testing brainfuck implementations using two code samples (bench.b and mandel.b).
Supports two mode:
- Verbose (default). Prints the output immediately.
- Quiet (if QUIET environment variable is set). Accumulates the output using Fletcher-16 checksum, and prints it out after the benchmark.
[Brainfuck](brainfuck)
### bench.b
| Language | Time, s | Memory, MiB | Energy, J |
| :--------------------- | -----------------------: | ------------------------------------------------: | -------------------------: |
| Scala (Staged) | 0.390±0.015 | 222.77±03.17 + 15.42±01.67 | 24.89±00.74 |
| Racket (Staged) | 0.893±0.001 | 101.38±00.97 + 0.00±00.00 | 34.61±00.07 |
| Rust | 1.011±0.000 | 1.05±00.01 + 0.00±00.00 | 43.04±00.28 |
| V/gcc | 1.060±0.000 | 1.92±00.03 + 0.00±00.00 | 43.67±00.15 |
| C/gcc | 1.111±0.001 | 1.01±00.01 + 0.00±00.00 | 47.01±00.65 |
| C++/g++ | 1.113±0.001 | 1.98±00.04 + 0.00±00.00 | 46.04±00.34 |
| D/gdc | 1.120±0.000 | 6.39±00.05 + 0.00±00.00 | 49.26±00.57 |
| C++/clang++ | 1.125±0.001 | 1.74±00.02 + 0.00±00.00 | 46.90±00.51 |
| C/clang | 1.140±0.001 | 1.02±00.01 + 0.00±00.00 | 48.57±00.37 |
| D/ldc2 | 1.163±0.003 | 2.32±00.80 + 0.00±00.00 | 49.08±00.30 |
| Nim/gcc | 1.168±0.001 | 1.01±00.02 + 0.00±00.00 | 49.66±00.25 |
| Java | 1.193±0.000 | 39.79±00.09 + 1.55±00.06 | 49.18±00.26 |
| Vala/gcc | 1.210±0.001 | 4.50±00.02 + 0.00±00.00 | 50.69±00.11 |
| Kotlin/JVM | 1.222±0.002 | 43.53±00.14 + 0.52±00.26 | 50.93±00.10 |
| Vala/clang | 1.238±0.006 | 4.52±00.03 + 0.00±00.00 | 52.25±01.41 |
| Go | 1.247±0.000 | 3.55±00.03 + 0.00±00.00 | 51.47±00.33 |
| Zig | 1.265±0.000 | 1.03±00.02 + 0.00±00.00 | 52.54±00.44 |
| C#/.NET Core | 1.367±0.001 | 32.66±00.13 + 0.15±00.02 | 58.23±00.27 |
| Go/gccgo | 1.481±0.002 | 23.56±00.01 + 0.00±00.00 | 62.86±00.67 |
| Nim/clang | 1.565±0.000 | 1.29±00.01 + 0.00±00.00 | 64.41±00.46 |
| F#/.NET Core | 1.574±0.004 | 37.41±00.18 + 0.30±00.00 | 67.12±00.40 |
| Crystal | 1.597±0.008 | 3.00±00.05 + 0.00±00.00 | 67.57±00.71 |
| OCaml | 1.638±0.005 | 3.01±00.05 + 2.81±00.06 | 76.01±00.28 |
| Julia | 1.638±0.003 | 256.65±00.11 + 0.40±00.03 | 69.57±00.34 |
| Chez Scheme | 1.714±0.004 | 24.77±00.02 + 4.46±00.12 | 72.48±00.24 |
| Racket | 1.756±0.028 | 113.87±00.09 + 0.00±00.00 | 71.30±01.18 |
| V/clang | 1.973±0.018 | 1.96±00.03 + 0.00±00.00 | 85.72±01.29 |
| C#/Mono | 2.054±0.011 | 25.60±00.08 + 0.00±00.00 | 87.72±01.03 |
| MLton | 2.088±0.027 | 1.77±00.04 + 0.25±00.00 | 86.67±01.18 |
| Scala | 2.779±0.003 | 72.03±00.11 + 249.04±00.15 | 121.64±00.71 |
| Node.js | 2.920±0.028 | 44.53±00.02 + 4.37±00.00 | 123.76±01.03 |
| Haskell (MArray) | 3.095±0.007 | 3.17±00.04 + 4.92±00.04 | 130.42±01.60 |
| D/dmd | 3.326±0.001 | 3.39±00.03 + 0.00±00.00 | 124.72±00.18 |
| Haskell (FP) | 3.624±0.002 | 3.27±00.05 + 4.89±00.04 | 154.06±00.80 |
| Ruby/truffleruby (JVM) | 4.996±0.396 | 374.17±10.09 + 496.92±17.09 | 243.84±22.97 |
| Ruby/truffleruby | 5.193±0.358 | 203.03±01.33 + 550.75±59.29 | 253.87±16.16 |
| Swift | 5.656±0.005 | 16.69±00.04 + 0.00±00.00 | 211.92±00.58 |
| Lua/luajit | 5.881±0.025 | 2.54±00.02 + 0.00±00.00 | 241.87±05.05 |
| Python/pypy | 9.544±0.044 | 58.79±00.12 + 29.96±00.01 | 422.77±04.07 |
| Idris | 15.129±0.051 | 20.71±00.03 + 8.83±00.04 | 661.53±10.66 |
| Elixir | 20.395±0.043 | 70.50±00.79 + 0.00±00.00 | 807.25±05.02 |
| Ruby (--jit) | 31.817±0.055 | 21.59±00.04 + 4.76±00.01 | 1311.22±06.23 |
| PHP | 34.492±0.160 | 17.78±00.18 + 0.00±00.00 | 1476.88±31.23 |
| Lua | 36.833±0.101 | 2.28±00.03 + 0.00±00.00 | 1512.87±19.03 |
| Ruby | 68.430±0.445 | 11.43±00.04 + 0.00±00.00 | 3008.91±26.07 |
| Python | 69.075±0.780 | 11.17±00.08 + 0.00±00.00 | 3039.83±50.68 |
| Ruby/jruby | 81.684±1.546 | 203.50±04.69 + 220.84±20.47 | 3624.22±70.81 |
| Tcl (FP) | 192.447±0.637 | 4.06±00.02 + 0.00±00.00 | 8478.25±83.42 |
| Perl | 225.331±1.241 | 6.90±00.05 + 0.00±00.00 | 10000.48±94.38 |
| Tcl (OOP) | 384.789±1.841 | 4.06±00.01 + 0.00±00.00 | 17091.07±181.64 |
### mandel.b
[Mandel in Brainfuck](brainfuck/mandel.b)
| Language | Time, s | Memory, MiB | Energy, J |
| :--------------------- | ----------------------: | ------------------------------------------------: | -----------------------: |
| Scala (Staged) | 7.639±0.120 | 224.42±01.90 + 102.36±04.76 | 459.89±14.99 |
| C++/g++ | 9.755±0.030 | 1.97±00.03 + 2.29±00.11 | 393.99±02.68 |
| C#/.NET Core | 12.048±0.028 | 32.72±00.06 + 1.27±00.00 | 482.24±01.22 |
| Java | 12.450±0.060 | 39.92±00.06 + 2.44±00.04 | 489.83±01.92 |
| C/gcc | 12.513±0.013 | 0.99±00.02 + 0.84±00.06 | 505.46±02.16 |
| Kotlin/JVM | 13.082±0.033 | 43.53±00.17 + 2.30±00.33 | 547.94±03.79 |
| F#/.NET Core | 13.115±0.017 | 37.55±00.07 + 2.12±00.04 | 526.18±01.55 |
| C/clang | 13.293±0.031 | 0.97±00.03 + 0.75±00.03 | 572.78±05.53 |
| V/gcc | 13.591±0.007 | 1.91±00.03 + 1.18±00.04 | 547.49±03.65 |
| C++/clang++ | 13.898±0.024 | 1.72±00.05 + 1.88±00.03 | 569.56±02.36 |
| Racket (Staged) | 13.986±0.059 | 99.66±00.50 + 77.13±01.67 | 556.40±01.66 |
| Crystal | 14.103±0.018 | 2.96±00.03 + 0.74±00.04 | 598.53±02.62 |
| Go | 14.182±0.009 | 3.56±00.01 + 0.00±00.00 | 562.34±00.49 |
| Rust | 14.229±0.012 | 1.04±00.01 + 1.20±00.04 | 564.72±02.02 |
| D/gdc | 14.266±0.016 | 6.37±00.06 + 1.49±00.04 | 601.20±03.61 |
| D/ldc2 | 14.278±0.013 | 3.05±00.02 + 0.83±00.02 | 572.27±01.12 |
| Vala/gcc | 14.469±0.022 | 4.47±00.04 + 1.24±00.04 | 574.39±01.98 |
| Zig | 14.481±0.014 | 1.02±00.02 + 1.42±00.06 | 600.92±02.27 |
| Vala/clang | 14.845±0.010 | 4.46±00.02 + 1.22±00.03 | 607.68±02.95 |
| Nim/gcc | 15.641±0.030 | 2.07±00.06 + 1.29±00.00 | 663.67±00.73 |
| Scala | 16.365±0.031 | 71.96±00.25 + 139.77±00.37 | 735.09±03.15 |
| Swift | 18.421±0.037 | 16.45±00.04 + 0.00±00.00 | 762.02±06.09 |
| Go/gccgo | 19.091±0.406 | 23.58±00.05 + 0.00±00.00 | 796.35±13.26 |
| Nim/clang | 19.696±0.204 | 2.38±00.01 + 1.29±00.00 | 806.37±09.58 |
| V/clang | 20.895±0.225 | 1.94±00.05 + 1.17±00.00 | 901.27±12.94 |
| OCaml | 24.390±0.016 | 4.53±00.08 + 2.82±00.00 | 1171.79±06.03 |
| Julia | 26.058±0.070 | 256.43±00.04 + 0.30±00.00 | 1021.32±06.88 |
| Chez Scheme | 27.698±0.056 | 25.55±00.06 + 3.68±00.01 | 1210.07±04.32 |
| C#/Mono | 31.025±0.011 | 25.61±00.09 + 0.83±00.00 | 1289.92±08.22 |
| MLton | 33.842±0.072 | 1.73±00.05 + 4.11±00.00 | 1537.44±17.63 |
| Haskell (MArray) | 34.546±0.022 | 4.08±00.02 + 5.11±00.00 | 1400.92±05.61 |
| Node.js | 34.598±0.063 | 44.44±00.07 + 5.25±00.00 | 1378.99±02.63 |
| Lua/luajit | 34.656±0.034 | 2.56±00.04 + 0.38±00.00 | 1388.13±02.21 |
| Racket | 35.183±0.294 | 113.90±00.09 + 1.45±00.65 | 1567.04±13.20 |
| D/dmd | 37.942±0.003 | 3.35±00.03 + 0.87±00.02 | 1378.02±00.92 |
| Python/pypy | 41.463±0.088 | 58.86±00.09 + 30.60±00.03 | 1837.98±07.63 |
| Ruby/truffleruby | 48.320±0.851 | 203.25±00.90 + 581.44±30.40 | 2360.50±62.00 |
| Ruby/truffleruby (JVM) | 50.546±1.177 | 374.73±03.49 + 489.39±89.50 | 2202.33±46.95 |
| Idris | 66.241±0.299 | 22.03±00.05 + 9.54±00.00 | 2848.93±05.21 |
| Haskell (FP) | 77.093±0.126 | 4.19±00.02 + 75.74±00.03 | 3195.67±11.41 |
## Base64
Testing base64 encoding/decoding of the large blob into the newly allocated buffers.
[Base64](base64)
| Language | Time, s | Memory, MiB | Energy, J |
| :------------------------ | ----------------------: | ------------------------------------------------: | ----------------------: |
| C/gcc (aklomp) | 0.098±0.000 | 2.20±00.03 + 0.00±00.00 | 4.60±00.03 |
| C/clang (aklomp) | 0.099±0.000 | 2.10±00.06 + 0.00±00.00 | 4.68±00.05 |
| PHP | 0.105±0.000 | 18.56±00.06 + 0.00±00.00 | 4.76±00.07 |
| Go (base64x) | 0.267±0.001 | 6.61±00.02 + 0.00±00.00 | 12.80±00.11 |
| Zig | 0.698±0.000 | 1.66±00.04 + 0.00±00.00 | 26.49±00.19 |
| Rust | 0.849±0.000 | 2.42±00.02 + 0.00±00.00 | 35.03±00.11 |
| Node.js | 0.913±0.001 | 43.17±00.03 + 40.43±00.21 | 40.38±00.37 |
| C/clang | 0.997±0.000 | 2.15±00.01 + 0.00±00.00 | 36.62±00.05 |
| C/gcc | 1.101±0.016 | 2.10±00.05 + 0.00±00.00 | 40.24±00.60 |
| Nim/clang | 1.103±0.001 | 2.06±00.02 + 5.79±00.03 | 44.95±00.27 |
| Crystal | 1.107±0.003 | 3.61±00.04 + 1.26±00.03 | 44.93±00.29 |
| D/ldc2 | 1.175±0.003 | 3.68±00.03 + 3.41±00.00 | 48.49±00.44 |
| Nim/gcc | 1.364±0.002 | 1.75±00.04 + 4.96±00.01 | 55.70±00.32 |
| Ruby (--jit) | 1.427±0.002 | 15.10±00.16 + 73.37±00.56 | 55.92±00.37 |
| Java | 1.518±0.005 | 41.05±00.10 + 210.65±19.06 | 60.66±00.44 |
| V/clang | 1.534±0.001 | 2.43±00.01 + 2386.75±01.65 | 57.81±00.29 |
| V/gcc | 1.575±0.000 | 2.43±00.03 + 2385.17±00.74 | 57.34±00.19 |
| Scala | 1.604±0.001 | 70.48±00.15 + 276.28±02.12 | 66.81±00.54 |
| Go | 1.622±0.002 | 4.34±00.02 + 0.00±00.00 | 67.66±00.22 |
| Ruby | 1.642±0.005 | 11.69±00.04 + 42.50±00.30 | 64.29±00.22 |
| Vala/clang | 1.643±0.001 | 5.68±00.03 + 0.10±00.03 | 63.09±00.70 |
| Vala/gcc | 1.644±0.001 | 5.66±00.01 + 0.08±00.05 | 62.92±00.40 |
| Kotlin/JVM | 1.665±0.003 | 44.48±00.20 + 314.13±03.17 | 67.36±00.44 |
| C++/g++ (libcrypto) | 1.711±0.003 | 6.05±00.16 + 0.76±00.03 | 68.53±00.38 |
| C++/clang++ (libcrypto) | 1.713±0.002 | 5.51±00.04 + 0.73±00.02 | 69.23±00.37 |
| Perl (MIME::Base64) | 1.863±0.044 | 14.57±00.04 + 0.12±00.03 | 75.20±01.07 |
| F#/.NET Core | 2.038±0.018 | 38.32±00.07 + 11.92±00.65 | 80.42±00.77 |
| C#/.NET Core | 2.181±0.021 | 33.77±00.03 + 12.54±02.09 | 85.47±00.28 |
| D/gdc | 2.382±0.001 | 7.44±00.04 + 3.36±00.00 | 103.50±00.94 |
| Go/gccgo | 2.949±0.007 | 24.81±00.17 + 0.00±00.00 | 139.67±00.50 |
| Julia | 3.079±0.006 | 271.96±00.08 + 97.24±00.20 | 124.20±01.58 |
| Python/pypy | 3.236±0.002 | 58.92±00.06 + 31.49±00.17 | 141.88±00.91 |
| D/dmd | 3.387±0.004 | 3.17±00.02 + 3.84±00.02 | 141.65±00.88 |
| Tcl | 3.438±0.001 | 5.20±00.02 + 0.00±00.00 | 137.53±00.49 |
| Python | 3.512±0.001 | 10.78±00.16 + 0.07±00.07 | 133.94±00.44 |
| Ruby/truffleruby (JVM) | 3.758±0.016 | 374.30±05.93 + 247.06±24.79 | 189.38±03.07 |
| Racket | 3.885±0.006 | 95.36±00.20 + 21.80±00.40 | 154.27±00.56 |
| C#/Mono | 4.632±0.005 | 26.39±00.04 + 18.66±00.02 | 190.60±01.36 |
| Ruby/jruby | 6.005±0.008 | 195.03±02.11 + 165.97±10.93 | 263.57±01.49 |
| Ruby/truffleruby | 8.324±0.018 | 201.01±02.82 + 554.96±13.20 | 400.06±01.94 |
| Perl (MIME::Base64::Perl) | 10.148±0.055 | 15.93±00.11 + 0.26±00.04 | 448.06±03.65 |
## Json
Testing parsing and simple calculating of values from a big JSON file.
Few notes:
- gason mutates input strings;
- simdjson requires input strings with batch of trailing zeros: a special zero padding for SIMD instructions;
- DAW JSON Link "NoCheck" skips some JSON structure correctness checks;
- DAW JSON Link, gason, default (not "Precise") RapidJSON, and D implementations except Mir-based
have some inaccuracies in number parsing:
- [DAW JSON Link's number parsing issue](https://github.com/beached/daw_json_link/issues/226)
- [gason's number parsing issue](https://github.com/vivkin/gason/issues/35)
- [D stdlib number parsing issue](https://issues.dlang.org/show_bug.cgi?id=20967)
[Json](json)
| Language | Time, s | Memory, MiB | Energy, J |
| :---------------------------------- | ----------------------: | --------------------------------------------------: | ----------------------: |
| C++/clang++ (simdjson On-Demand) | 0.060±0.000 | 112.46±00.06 + 60.05±00.06 | 2.51±00.01 |
| C++/g++ (simdjson On-Demand) | 0.061±0.000 | 113.66±00.03 + 59.81±00.00 | 2.57±00.02 |
| C++/clang++ (DAW JSON Link NoCheck) | 0.083±0.000 | 112.54±00.05 + 0.00±00.00 | 3.37±00.03 |
| C++/g++ (DAW JSON Link NoCheck) | 0.087±0.000 | 113.29±00.06 + 0.00±00.00 | 3.53±00.04 |
| C++/clang++ (DAW JSON Link) | 0.093±0.000 | 112.47±00.03 + 0.00±00.00 | 3.90±00.07 |
| C++/g++ (DAW JSON Link) | 0.093±0.000 | 113.30±00.05 + 0.00±00.00 | 3.88±00.03 |
| Rust (Serde Custom) | 0.098±0.001 | 111.40±00.08 + 0.00±00.00 | 4.14±00.04 |
| C++/clang++ (simdjson DOM) | 0.102±0.000 | 112.55±00.09 + 177.09±00.06 | 4.62±00.02 |
| C++/g++ (simdjson DOM) | 0.108±0.000 | 113.54±00.06 + 172.73±00.13 | 4.90±00.02 |
| Rust (Serde Typed) | 0.111±0.000 | 111.71±00.05 + 11.05±00.07 | 4.62±00.04 |
| D/ldc2 (Mir Asdf DOM) | 0.131±0.000 | 112.93±00.04 + 61.22±00.00 | 5.43±00.05 |
| C++/clang++ (gason) | 0.139±0.000 | 112.57±00.02 + 96.97±00.06 | 5.67±00.04 |
| C++/g++ (gason) | 0.140±0.000 | 113.17±00.06 + 96.93±00.04 | 5.55±00.03 |
| C++/g++ (RapidJSON) | 0.152±0.000 | 113.21±00.04 + 128.90±00.03 | 6.47±00.04 |
| Scala (jsoniter-scala) | 0.157±0.002 | 290.75±00.24 + 20.43±00.24 | 8.36±00.07 |
| Go (rjson custom) | 0.196±0.000 | 113.58±00.05 + 0.00±00.00 | 7.57±00.02 |
| C++/clang++ (RapidJSON) | 0.200±0.000 | 112.47±00.04 + 128.91±00.09 | 8.55±00.08 |
| Go (Sonic) | 0.208±0.002 | 122.51±00.12 + 0.00±00.00 | 8.94±00.09 |
| C++/g++ (RapidJSON Precise) | 0.217±0.000 | 113.21±00.06 + 122.47±01.29 | 9.28±00.08 |
| D/ldc2 (Mir Amazon's Ion DOM) | 0.219±0.000 | 113.03±00.08 + 80.70±00.00 | 9.26±00.03 |
| Go (rjson) | 0.224±0.000 | 113.61±00.10 + 0.00±00.00 | 8.70±00.02 |
| Zig | 0.241±0.000 | 111.03±00.02 + 39.25±00.00 | 10.37±00.07 |
| Go (goccy/go-json) | 0.266±0.000 | 114.29±00.07 + 0.00±00.00 | 10.56±00.02 |
| C++/clang++ (RapidJSON Precise) | 0.285±0.000 | 112.48±00.03 + 128.91±00.03 | 12.50±00.10 |
| C++/g++ (RapidJSON SAX) | 0.346±0.001 | 113.11±00.08 + 0.00±00.00 | 15.32±00.06 |
| C/clang (yajl) | 0.350±0.001 | 110.97±00.04 + 0.00±00.00 | 15.14±00.06 |
| C/gcc (yajl) | 0.353±0.001 | 110.95±00.01 + 0.00±00.00 | 15.22±00.06 |
| C++/g++ (Boost.JSON) | 0.363±0.000 | 113.32±00.04 + 308.09±00.02 | 15.44±00.10 |
| C++/clang++ (Boost.JSON) | 0.371±0.001 | 112.68±00.03 + 308.09±00.03 | 15.74±00.10 |
| Nim/clang (jsony) | 0.392±0.000 | 111.56±00.05 + 146.12±00.09 | 16.46±00.12 |
| C++/g++ (RapidJSON SAX Precise) | 0.401±0.001 | 113.12±00.06 + 0.00±00.00 | 17.92±00.23 |
| C++/clang++ (RapidJSON SAX) | 0.408±0.001 | 194.77±00.07 + 0.00±00.00 | 17.40±00.07 |
| Nim/gcc (jsony) | 0.412±0.000 | 111.23±00.03 + 154.69±00.03 | 17.46±00.15 |
| Node.js | 0.470±0.004 | 152.69±00.04 + 195.96±00.68 | 22.05±00.12 |
| C++/clang++ (RapidJSON SAX Precise) | 0.489±0.001 | 194.79±00.07 + 0.00±00.00 | 21.84±00.23 |
| Go (jsoniter) | 0.513±0.000 | 114.27±00.06 + 0.00±00.00 | 20.75±00.17 |
| Rust (Serde Untyped) | 0.532±0.001 | 111.73±00.03 + 839.98±00.00 | 22.14±00.20 |
| C#/.NET Core (System.Text.Json) | 0.543±0.002 | 489.53±00.11 + 140.62±00.22 | 24.07±00.23 |
| Java (DSL-JSON) | 0.568±0.007 | 262.39±00.11 + 225.35±19.01 | 29.30±00.32 |
| Python/pypy | 0.608±0.002 | 279.98±00.08 + 125.71±00.04 | 26.46±00.26 |
| V/gcc | 0.615±0.001 | 111.47±00.04 + 496.21±00.00 | 25.70±00.23 |
| V/clang | 0.637±0.001 | 111.87±00.01 + 495.83±00.00 | 26.74±00.19 |
| Nim/gcc (Packedjson) | 0.638±0.003 | 111.93±00.02 + 294.16±00.00 | 27.28±00.40 |
| Crystal (Pull) | 0.638±0.002 | 113.26±00.02 + 18.44±00.00 | 28.10±00.40 |
| Crystal (Schema) | 0.656±0.001 | 113.23±00.03 + 48.86±00.13 | 28.63±00.07 |
| Nim/clang (Packedjson) | 0.662±0.001 | 112.23±00.01 + 294.16±00.00 | 28.31±00.17 |
| Perl (Cpanel::JSON::XS) | 0.765±0.003 | 125.17±00.05 + 402.77±00.00 | 31.84±00.14 |
| Go | 0.813±0.001 | 113.95±00.06 + 0.00±00.00 | 33.60±00.06 |
| PHP | 0.813±0.001 | 128.26±00.08 + 517.83±00.03 | 34.57±00.20 |
| Crystal | 0.924±0.004 | 113.21±00.02 + 392.50±00.03 | 39.68±00.33 |
| Nim/gcc | 1.037±0.002 | 111.91±00.03 + 1001.34±00.00 | 43.13±00.12 |
| C#/.NET Core | 1.056±0.004 | 495.03±00.30 + 272.99±00.16 | 50.49±00.23 |
| Nim/clang | 1.076±0.002 | 112.23±00.00 + 999.02±00.00 | 44.69±00.27 |
| C++/g++ (json-c) | 1.159±0.003 | 113.42±00.05 + 1216.05±00.01 | 47.99±00.30 |
| C++/clang++ (json-c) | 1.160±0.007 | 112.70±00.07 + 1216.04±00.01 | 48.35±00.66 |
| Clojure | 1.161±0.019 | 452.13±02.50 + 587.46±09.25 | 61.05±00.99 |
| C++/clang++ (Nlohmann) | 1.176±0.001 | 112.64±00.03 + 360.11±00.02 | 50.39±00.68 |
| Ruby (--jit) | 1.280±0.005 | 127.59±00.08 + 212.39±00.01 | 53.84±00.51 |
| Go/gccgo | 1.290±0.004 | 138.88±00.10 + 0.00±00.00 | 53.03±00.36 |
| C++/g++ (Nlohmann) | 1.327±0.003 | 113.27±00.06 + 448.05±00.01 | 56.39±00.35 |
| Ruby | 1.328±0.006 | 121.40±00.05 + 212.77±00.02 | 55.73±00.73 |
| CPython (UltraJSON) | 1.331±0.004 | 123.14±00.05 + 497.28±01.55 | 51.19±00.22 |
| Python | 1.374±0.002 | 120.99±00.03 + 325.95±00.01 | 55.23±00.37 |
| F#/.NET Core (System.Text.Json) | 1.496±0.003 | 498.18±00.08 + 232.75±04.63 | 68.50±00.69 |
| Ruby (YAJL) | 1.719±0.009 | 121.49±00.11 + 218.39±00.03 | 72.78±00.52 |
| D/ldc2 | 1.723±0.003 | 112.86±00.06 + 708.76±00.04 | 72.08±00.65 |
| C#/Mono | 1.809±0.013 | 252.32±00.17 + 31.51±00.01 | 78.45±00.96 |
| Haskell | 1.982±0.007 | 115.63±00.14 + 723.73±00.48 | 84.53±00.77 |
| Rust (jq) | 2.529±0.003 | 113.39±00.06 + 902.98±01.54 | 106.08±00.37 |
| C++/g++ (Boost.PropertyTree) | 2.611±0.008 | 113.14±00.04 + 1440.09±00.04 | 111.72±00.36 |
| C++/clang++ (Boost.PropertyTree) | 2.671±0.005 | 195.01±00.04 + 1232.80±00.03 | 112.86±00.70 |
| Ruby/jruby | 2.884±0.022 | 464.98±06.74 + 972.45±49.49 | 148.64±02.99 |
| D/dmd | 3.073±0.003 | 113.16±00.04 + 708.78±00.07 | 131.07±00.23 |
| Vala/clang | 3.165±0.007 | 115.04±00.04 + 980.08±00.03 | 137.65±00.31 |
| Vala/gcc | 3.166±0.011 | 115.06±00.03 + 980.07±00.03 | 136.93±00.87 |
| D/gdc | 3.569±0.004 | 116.68±00.06 + 680.99±00.12 | 151.81±01.75 |
| Racket | 3.818±0.022 | 222.34±00.45 + 261.38±27.93 | 159.23±01.29 |
| Perl (JSON::Tiny) | 9.050±0.121 | 125.59±00.05 + 528.92±00.02 | 394.96±03.88 |
| Ruby/truffleruby | 10.718±0.048 | 522.13±27.57 + 1942.99±149.89 | 617.74±02.74 |
| Ruby/truffleruby (JVM) | 11.011±0.153 | 474.89±08.90 + 2201.50±87.97 | 693.97±13.20 |
## Matmul
Testing allocating and multiplying matrices.
[Matmul](matmul)
| Language | Time, s | Memory, MiB | Energy, J |
| :---------------------- | ------------------------: | -------------------------------------------------: | -------------------------: |
| D/ldc2 (lubeck) | 0.032±0.001 | 40.38±00.48 + 23.61±00.41 | 3.61±00.09 |
| Nim/gcc (Arraymancer) | 0.064±0.004 | 5.41±00.06 + 57.41±00.10 | 5.27±00.20 |
| Python (NumPy) | 0.065±0.000 | 33.37±00.06 + 57.87±00.04 | 5.54±00.01 |
| Java (ND4J) | 0.076±0.001 | 114.91±01.04 + 92.22±00.01 | 5.97±00.05 |
| Rust (ndarray) | 0.089±0.001 | 2.50±00.04 + 68.53±00.00 | 6.03±00.08 |
| Julia (threads: 2) | 0.092±0.000 | 300.38±00.45 + 47.53±00.53 | 5.54±00.03 |
| Nim/clang (Arraymancer) | 0.128±0.020 | 6.05±00.18 + 57.54±00.17 | 8.78±01.10 |
| Julia (threads: 1) | 0.142±0.000 | 299.90±00.18 + 47.76±00.23 | 6.91±00.04 |
| C++/g++ (Eigen) | 0.145±0.000 | 4.49±00.06 + 85.26±00.00 | 7.14±00.03 |
| C++/clang++ (Eigen) | 0.145±0.000 | 4.72±00.03 + 85.37±00.00 | 7.07±00.04 |
| V/clang (VSL + CBLAS) | 0.254±0.004 | 7.13±00.03 + 51.89±00.00 | 17.98±00.26 |
| V/clang (VSL) | 0.258±0.002 | 7.21±00.17 + 51.57±00.13 | 17.89±00.11 |
| V/gcc (VSL + CBLAS) | 0.459±0.002 | 7.25±00.06 + 51.57±00.00 | 34.63±00.06 |
| V/gcc (VSL) | 0.466±0.001 | 6.90±00.04 + 51.89±00.00 | 31.75±00.07 |
| Julia (no BLAS) | 1.146±0.023 | 271.40±00.23 + 52.30±00.01 | 49.02±00.55 |
| D/ldc2 | 1.714±0.001 | 3.43±00.04 + 70.44±00.00 | 62.98±00.24 |
| D/gdc | 1.866±0.001 | 7.40±00.14 + 70.16±00.01 | 72.94±00.11 |
| D/dmd | 1.874±0.001 | 3.32±00.03 + 70.46±00.03 | 70.33±00.49 |
| C/gcc | 3.025±0.000 | 1.53±00.02 + 68.65±00.02 | 109.57±01.18 |
| V/gcc | 3.027±0.001 | 2.60±00.03 + 68.58±00.00 | 112.33±00.33 |
| Vala/clang | 3.057±0.000 | 5.52±00.03 + 68.32±00.00 | 104.60±00.31 |
| V/clang | 3.058±0.000 | 2.90±00.06 + 68.58±00.00 | 104.38±00.16 |
| C/clang | 3.060±0.000 | 1.55±00.02 + 68.63±00.04 | 104.48±00.15 |
| Rust | 3.061±0.000 | 2.14±00.07 + 68.69±00.00 | 105.19±00.52 |
| Zig | 3.063±0.001 | 1.92±00.01 + 68.58±00.00 | 108.31±00.18 |
| Nim/gcc | 3.086±0.001 | 2.60±00.01 + 58.91±01.16 | 114.12±00.58 |
| Swift | 3.090±0.000 | 7.99±00.06 + 68.69±00.00 | 109.94±00.59 |
| Nim/clang | 3.116±0.001 | 2.88±00.01 + 57.75±00.00 | 107.30±00.87 |
| Vala/gcc | 3.123±0.000 | 5.49±00.10 + 68.32±00.00 | 114.09±00.15 |
| Go | 3.145±0.000 | 4.09±00.01 + 0.00±00.00 | 112.66±01.59 |
| Crystal | 3.147±0.001 | 3.63±00.06 + 60.02±00.02 | 115.55±00.61 |
| Go/gccgo | 3.150±0.000 | 24.19±00.16 + 0.00±00.00 | 110.47±00.13 |
| Java | 3.165±0.002 | 40.98±00.16 + 74.30±00.20 | 117.37±01.36 |
| Kotlin/JVM | 3.209±0.003 | 42.32±00.21 + 73.03±00.22 | 129.95±00.33 |
| Node.js | 3.252±0.002 | 51.38±00.26 + 70.87±00.47 | 130.78±00.37 |
| Python/pypy | 3.254±0.001 | 59.96±00.07 + 68.90±00.03 | 135.03±00.17 |
| Scala | 3.332±0.003 | 71.86±00.08 + 153.79±00.24 | 120.95±00.68 |
| C#/.NET Core | 4.893±0.001 | 34.69±00.07 + 68.91±00.03 | 196.73±00.66 |
| C#/Mono | 7.391±0.000 | 26.02±00.07 + 69.48±00.01 | 297.35±00.51 |
| Ruby/truffleruby | 18.156±3.646 | 360.35±12.57 + 574.17±41.24 | 647.87±107.81 |
| Ruby/truffleruby (JVM) | 25.014±0.847 | 407.30±16.57 + 334.68±58.11 | 875.73±22.67 |
| Perl | 149.225±1.755 | 8.52±00.04 + 379.63±00.02 | 6447.37±59.99 |
| Python | 155.211±1.106 | 10.91±00.03 + 68.84±00.00 | 7060.84±46.09 |
| Ruby (--jit) | 156.368±0.039 | 21.96±00.04 + 68.35±00.02 | 6925.51±35.94 |
| Ruby | 195.516±0.462 | 11.75±00.07 + 69.19±00.01 | 8795.03±25.92 |
| Tcl | 202.975±0.793 | 7.47±00.02 + 400.44±00.00 | 9311.02±86.81 |
| Ruby/jruby | 361.315±19.249 | 273.70±12.97 + 1142.23±73.73 | 14778.53±653.23 |
## Primes
Testing:
- generating primes using the optimized [sieve of Atkin](https://www.geeksforgeeks.org/sieve-of-atkin/);
- prefix search for their decimal numbers using Trie data structure.
Notes:
- All languages but V and Python use unordered hashmaps (V and Python don't provide those out of box, and
their hashmaps use keys in the insertion order);
- The results are always sorted (could be unstable or stable though).
[Primes](primes)
| Language | Time, s | Memory, MiB | Energy, J |
| :--------------------- | ---------------------: | ------------------------------------------------: | ----------------------: |
| Zig | 0.056±0.000 | 1.03±00.02 + 52.80±00.19 | 2.34±00.03 |
| C++/clang++ | 0.062±0.000 | 3.17±00.03 + 54.80±00.00 | 2.38±00.02 |
| C++/g++ | 0.063±0.000 | 3.67±00.07 + 70.55±00.25 | 2.44±00.02 |
| Go | 0.072±0.000 | 3.08±00.05 + 0.00±00.00 | 3.06±00.04 |
| V/clang | 0.095±0.000 | 2.14±00.23 + 211.15±00.64 | 3.91±00.04 |
| V/gcc | 0.099±0.000 | 1.95±00.03 + 201.60±00.17 | 4.10±00.03 |
| Rust | 0.101±0.000 | 2.09±00.11 + 74.71±00.77 | 3.98±00.05 |
| Java | 0.130±0.004 | 39.84±00.17 + 123.02±05.25 | 7.33±00.26 |
| Crystal | 0.138±0.000 | 3.71±00.02 + 88.43±00.00 | 5.58±00.05 |
| Node.js | 0.218±0.002 | 42.17±00.00 + 147.09±02.74 | 10.79±00.07 |
| Scala | 0.228±0.003 | 72.13±00.08 + 230.29±00.88 | 13.20±00.22 |
| Nim/clang | 0.278±0.000 | 2.10±00.10 + 587.68±01.03 | 10.75±00.04 |
| Nim/gcc | 0.287±0.000 | 1.90±00.02 + 615.91±00.00 | 10.83±00.07 |
| Lua/luajit | 0.298±0.001 | 1.33±00.02 + 156.66±00.63 | 12.00±00.08 |
| Python/pypy | 0.616±0.003 | 58.72±00.13 + 249.30±00.15 | 25.12±00.16 |
| Julia | 0.651±0.001 | 271.49±00.14 + 370.21±01.14 | 25.06±00.21 |
| Racket | 0.752±0.002 | 109.49±00.82 + 247.11±00.45 | 29.59±00.17 |
| Ruby/truffleruby | 0.890±0.012 | 203.57±01.55 + 800.19±11.11 | 60.49±00.76 |
| Lua | 1.142±0.006 | 2.69±00.04 + 283.50±01.03 | 46.78±00.35 |
| Ruby (--jit) | 1.187±0.002 | 22.98±00.02 + 163.71±00.14 | 47.95±00.43 |
| Ruby/truffleruby (JVM) | 1.378±0.088 | 376.75±09.21 + 495.42±42.25 | 89.49±05.72 |
| Ruby | 1.860±0.003 | 11.47±00.07 + 172.19±00.35 | 76.75±00.60 |
| Ruby/jruby | 2.314±0.075 | 197.13±08.38 + 521.12±52.24 | 124.24±05.36 |
| Python | 2.530±0.014 | 10.84±00.04 + 181.69±01.57 | 107.30±01.03 |
# Tests Execution
## Environment
CPU: Intel(R) Xeon(R) E-2324G
Base Docker image: Debian GNU/Linux bookworm/sid
| Language | Version |
| ---------------- | ------------------------------- |
| .NET Core | 8.0.300 |
| C#/.NET Core | 4.10.0-3.24216.12 (3af0081a) |
| C#/Mono | 6.12.0.200 |
| Chez Scheme | 9.5.8 |
| Clojure | "1.11.3" |
| Crystal | 1.12.1 |
| D/dmd | v2.108.1 |
| D/gdc | 13.2.0 |
| D/ldc2 | 1.38.0 |
| Elixir | 1.14.0 |
| F#/.NET Core | 12.8.300.0 for F# 8.0 |
| Go | go1.22.3 |
| Go/gccgo | 13.2.0 |
| Haskell | 9.8.2 |
| Idris 2 | 0.6.0 |
| Java | 22.0.1 |
| Julia | v"1.10.3" |
| Kotlin | 2.0.0 |
| Lua | 5.4.6 |
| Lua/luajit | 2.1.1710398010 |
| MLton | 20210117 |
| Nim | 2.0.4 |
| Node.js | v22.2.0 |
| OCaml | 5.2.0 |
| PHP | 8.2.18 |
| Perl | v5.38.2 |
| Python | 3.11.9 |
| Python/pypy | 7.3.16-final0 for Python 3.10.14 |
| Racket | "8.13" |
| Ruby | 3.3.1p55 |
| Ruby/jruby | 9.4.7.0 |
| Ruby/truffleruby | 24.0.1 |
| Rust | 1.78.0 |
| Scala | 3.4.2 |
| Swift | 5.10 |
| Tcl | 8.6 |
| V | 0.4.6 736067d |
| Vala | 0.56.17 |
| Zig | 0.12.0 |
| clang/clang++ | 16.0.6 (27) |
| gcc/g++ | 13.2.0 |
## Using Docker
Build the image:
$ docker build docker/ -t benchmarks
Run the image:
$ docker run -it --rm -v $(pwd):/src benchmarks
where `` is:
- `versions` (print installed language versions);
- `shell` (start the shell);
- `brainfuck bench` (build and run Brainfuck bench.b benchmarks);
- `brainfuck mandel` (build and run Brainfuck mandel.b benchmarks);
- `base64` (build and run Base64 benchmarks);
- `json` (build and run Json benchmarks);
- `matmul` (build and run Matmul benchmarks);
- `primes` (build and run Primes benchmarks);
Please note that the actual measurements provided in the project are taken semi-manually (via `shell`) as the full update takes days and could have occassional issues in Docker.
There is a `./run.sh` that could be used to simplify Docker usage:
- `./run.sh build` (build the image);
- `./run.sh make versions` (run the image with the `versions` command);
- `sudo ./run.sh shell` (run the image with the `shell' command, sudo is required to read energy levels).
## Manual Execution
Makefiles contain recipes for building and executing tests with the
proper dependencies. Please use `make run` (and `make run2` where applicable).
The measurements are taken using `analyze.rb` script:
$ cd
$ ../analyze.rb make run
$ ../analyze.rb make run[]
Please note that the measurements could take hours. It uses 10 iterations
by default, but it could be changed using ATTEMPTS environment variable:
$ ATTEMPTS=1 ../analyze.rb make run
### Prerequisites
Please use [Dockerfile](docker/Dockerfile) as a reference regarding which
packages and tools are required.
For all (optional):
- [Powercap](https://github.com/powercap/powercap) for reading energy
counters in Linux (Debian package `powercap-utils`).
For Python:
- [NumPy](https://numpy.org/) for matmul tests
(Debian package `python3-numpy`).
- [UltraJSON](https://pypi.org/project/ujson/) for JSON tests
(Debian package `python3-ujson`).
For C++:
- [Boost](https://www.boost.org/) for JSON tests
(Debian package `libboost-dev`).
- [JSON-C](https://github.com/json-c/json-c) for JSON tests
(Debian package `libjson-c-dev`).
For Rust:
- [libjq](https://stedolan.github.io/jq/) for jq test
(Debian packages `libjq-dev`, `libonig-dev` and environment variable
`JQ_LIB_DIR=/usr/lib/x86_64-linux-gnu/`).
For Java, Scala:
- [Coursier](https://get-coursier.io/) for downloading Maven artifacts.
For Haskell:
- [network](http://hackage.haskell.org/package/network) for
TCP connectivity between the tests and the test runner.
- [raw-strings-qq](http://hackage.haskell.org/package/raw-strings-qq) for
raw string literals used in tests.
For Perl:
- [cpanminus](https://metacpan.org/pod/App::cpanminus) for installing
modules from CPAN (Debian package `cpanminus`).
For Vala:
- [JSON-GLib](https://wiki.gnome.org/Projects/JsonGlib) for JSON tests
(Debian package `libjson-glib-dev`).
# Contribution
Please follow the criteria specified in the [overview](#overview). Besides
that please ensure that the communication protocol between a test and the
test runner is satisfied:
- The test runner listens on localhost:9001;
- All messages are sent using TCP sockets closed immediately after the
message has been sent;
- There are two messages sent from a test (it establishes the measurement
boundary):
1. The beginning message having the format *name of the test*/t*process ID*
(the process ID is used to measure the memory consumption). Please note that
the name of the test couldn't use Tab character as it's a delimiter;
2. The end message with any content (mostly it's "stop" for consistency).
- The test runner could be unavailable (if the test is launched as is) and
the test should gracefully handle it.
## Makefile guide
### Binary executables
If the test is compiled into a single binary, then two sections of
the `Makefile` require changes:
- append a new target (the final binary location) into `executables`
variable;
- append the proper target rule.
### Compiled artifacts
If the test is compiled, but can't be executed directly as a binary, then
three sections of the `Makefile` require changes:
- append a new target (the final artifact location) into `artifacts`
variable;
- append the proper target rule to compile the test;
- append `run[]` rule to run the test.
### Scripting language
If the test doesn't require compilation, then two sections of the `Makefile`
requires changes:
- append `run[]` into `all_runners` variable;
- append `run[]` rule to run the test.
## README update
TOC is regenerated using [git-markdown-toc](https://github.com/ildar-shaimordanov/git-markdown-toc):
```
./run.sh toc
```