Last year, we've done string associative benchmark and lesser string associative benchmark (measuring string concat operation and built-in associative array set and get), numeric comb sort benchmark and string comb sort benchmark (measuring
basic array random access, string conversion, and array swap for number
and string), this year's using newer server: 32-core
running on 64-bit Ubuntu 21.10. This time we will skip programming languages that are no deb packages (unless the install script is just one line and doesn't ruin system directories) or no direct compile-run command like previous one, also only best of 3 runs.
$ alias time='/usr/bin/time -f "\nCPU: %Us\tReal: %es\tRAM: %MKB"
This time, NodeJS failed to complete (after waiting for an hour) with 10x more data compared to last year for assoc benchmark. Here's the spreadsheet and final result (Real duration and RAM):
| Language | Command Flags | Version | Assoc | RAM | Num Comb | RAM | Str Comb | RAM | Total | RAM |
| Go | go run | go1.17.5 | 12.39 | 2,305,824 | 0.46 | 83,096 | 3.33 | 245,896 | 16.18 | 2,634,816 |
| Java | java | 18-ea+15-Ubuntu-4 | 10.70 | 5,582,308 | 0.96 | 170,240 | 6.93 | 722,152 | 18.59 | 6,474,700 |
| Nim | nim r -d:release --gc:arc | 1.4.2 | 17.90 | 4,200,212 | 1.21 | 79,444 | 5.45 | 633,816 | 24.56 | 4,913,472 |
| Python | pypy | 7.3.5 | 19.70 | 3,104,440 | 2.17 | 140,124 | 5.73 | 523,264 | 27.60 | 3,767,828 |
| C | tcc -run | 0.9.27 | 22.68 | 2,820,568 | 1.01 | 80,484 | 4.80 | 392,896 | 28.49 | 3,293,948 |
| Julia | julia | 1.6.5 | 23.98 | 3,714,448 | 0.52 | 255,844 | 4.60 | 861,076 | 29.10 | 4,831,368 |
| V | v -prod run | 0.2.4 a0a1807 | 20.36 | 8,910,856 | 2.67 | 124,248 | 6.48 | 470,136 | 29.51 | 9,505,240 |
| Lua | luajit | 2.1.0-beta3 | 16.70 | 1,133,876 | 3.27 | 133,504 | 10.80 | 511,468 | 30.77 | 1,778,848 |
| Dart | dart | 2.15.1 | 25.92 | 2,101,152 | 1.40 | 208,056 | 5.75 | 574,852 | 33.07 | 2,884,060 |
| Crystal | crystal run --release | 1.2.2 | 13.77 | 2,371,320 | 7.73 | 202,552 | 12.08 | 440,748 | 33.58 | 3,014,620 |
| Nim | nim r -d:release | 1.4.2 | 24.92 | 3,864,100 | 1.76 | 79,548 | 7.51 | 1,211,200 | 34.19 | 5,154,848 |
| PHP | php | 8.0.8 | 9.94 | 1,368,808 | 7.74 | 328,344 | 24.84 | 641,448 | 42.52 | 2,338,600 |
| Crystal | crystal run | 0.35.1 | 38.85 | 2,372,004 | 9.97 | 179,176 | 22.27 | 441,720 | 71.09 | 2,992,900 |
| V | v run | 0.2.4 a0a1807 | 51.81 | 8,911,004 | 6.63 | 79,716 | 18.43 | 470,420 | 76.87 | 9,461,140 |
| Python | python3 | 3.9.7 | 43.11 | 4,106,892 | 29.16 | 405,332 | 43.08 | 722,996 | 115.35 | 5,235,220 |
| Nim | nim r | 1.4.2 | 88.05 | 3,864,048 | 2.93 | 79,536 | 32.60 | 1,211,260 | 123.58 | 5,154,844 |
| Ruby | ruby | ruby 2.7.4p191 | 52.48 | 2,970,908 | 27.15 | 100,320 | 52.23 | 708,940 | 131.86 | 3,780,168 |
| Javascript | node | v16.13.1 | 999.99 | 9,999,999 | 0.71 | 115,044 | 6.11 | 461,836 | 1006.81 | 10,576,879 |
FAQ
1. Why you measure the compile duration too? because developer experience also important (feedback loop, edit-rebuild/compile-run), at least for me, it would be sucks a lot if we have to wait a minute to compile before we can test something. We could always write precalculated values with C++ template to make runtime faster for example, but the compilation delay would be very sucks.
2. Why not warming up the VM first? each implementation have it's own advantage and disadvantage. We already know, that compiled language mostly faster at runtime, but at cost of relatively slower development feedback loop. Interpreted language mostly slower at runtime, especially if executed using VM that have startup overhead, in exception of one with AOT or JIT optimization. So to make it fair for every kind of implementation, we do it differently by not glorifying the runtime performance (which super make sense for server or long-lived process, but not best for development or CI cost which people often neglected), but by total performance which consist of Compile duration (if any) + VM startup duration (if any) + AOT or JIT duration (if any) + Runtime duration, so every strategy the PL's implementator use can be fairly judged.
3.
Why there's no C++, VB.NET, C#, D, Object-Pascal? don't want to compile
things (since there's no build and run command in one flag).
4. Why there's no Kotlin, Scala, Rust, Elixir, Pony, Swift, Groovy, or Zig? Too lazy to add :3 you can
contribute tho
(create a pull request, then I'll run the benchmark again as
preferably as there's precompiled binary/deb/apt/ppa repository for the
compiler/interpreter).
5. Why there's no Ruby 3.1? I can't find any PPA for latest Ruby, latest one on Ubuntu 21.10 repo is Ruby2.7.
Contributors: ilmanzo (Nim, Crystal, D), inkydragon (Julia)
