String Associative Array and CombSort Benchmark 2021 Edition

Last year, we've done string associative benchmark and lesser string associative benchmark (measuring string concat operation and built-in associative array set and get), numeric comb sort benchmark and string comb sort benchmark (measuring basic array random access, string conversion, and array swap for number and string), this year's using newer server: 32-core running on 64-bit Ubuntu 21.10. This time we will skip programming languages that are no deb packages (unless the install script is just one line and doesn't ruin system directories) or no direct compile-run command like previous one, also only best of 3 runs.

$ alias time='/usr/bin/time -f "\nCPU: %Us\tReal: %es\tRAM: %MKB"

This time, NodeJS failed to complete (after waiting for an hour) with 10x more data compared to last year for assoc benchmark. Here's the spreadsheet and final result (Real duration and RAM):

Language	Command Flags	Version	Assoc	RAM	Num Comb	RAM	Str Comb	RAM	Total	RAM
Go	go run	go1.17.5	12.39	2,305,824	0.46	83,096	3.33	245,896	16.18	2,634,816
Java	java	18-ea+15-Ubuntu-4	10.70	5,582,308	0.96	170,240	6.93	722,152	18.59	6,474,700
Nim	nim r -d:release --gc:arc	1.4.2	17.90	4,200,212	1.21	79,444	5.45	633,816	24.56	4,913,472
Python	pypy	7.3.5	19.70	3,104,440	2.17	140,124	5.73	523,264	27.60	3,767,828
C	tcc -run	0.9.27	22.68	2,820,568	1.01	80,484	4.80	392,896	28.49	3,293,948
Julia	julia	1.6.5	23.98	3,714,448	0.52	255,844	4.60	861,076	29.10	4,831,368
V	v -prod run	0.2.4 a0a1807	20.36	8,910,856	2.67	124,248	6.48	470,136	29.51	9,505,240
Lua	luajit	2.1.0-beta3	16.70	1,133,876	3.27	133,504	10.80	511,468	30.77	1,778,848
Dart	dart	2.15.1	25.92	2,101,152	1.40	208,056	5.75	574,852	33.07	2,884,060
Crystal	crystal run --release	1.2.2	13.77	2,371,320	7.73	202,552	12.08	440,748	33.58	3,014,620
Nim	nim r -d:release	1.4.2	24.92	3,864,100	1.76	79,548	7.51	1,211,200	34.19	5,154,848
PHP	php	8.0.8	9.94	1,368,808	7.74	328,344	24.84	641,448	42.52	2,338,600
Crystal	crystal run	0.35.1	38.85	2,372,004	9.97	179,176	22.27	441,720	71.09	2,992,900
V	v run	0.2.4 a0a1807	51.81	8,911,004	6.63	79,716	18.43	470,420	76.87	9,461,140
Python	python3	3.9.7	43.11	4,106,892	29.16	405,332	43.08	722,996	115.35	5,235,220
Nim	nim r	1.4.2	88.05	3,864,048	2.93	79,536	32.60	1,211,260	123.58	5,154,844
Ruby	ruby	ruby 2.7.4p191	52.48	2,970,908	27.15	100,320	52.23	708,940	131.86	3,780,168
Javascript	node	v16.13.1	999.99	9,999,999	0.71	115,044	6.11	461,836	1006.81	10,576,879

FAQ

1. Why you measure the compile duration too? because developer experience also important (feedback loop, edit-rebuild/compile-run), at least for me, it would be sucks a lot if we have to wait a minute to compile before we can test something. We could always write precalculated values with C++ template to make runtime faster for example, but the compilation delay would be very sucks.

2. Why not warming up the VM first? each implementation have it's own advantage and disadvantage. We already know, that compiled language mostly faster at runtime, but at cost of relatively slower development feedback loop. Interpreted language mostly slower at runtime, especially if executed using VM that have startup overhead, in exception of one with AOT or JIT optimization. So to make it fair for every kind of implementation, we do it differently by not glorifying the runtime performance (which super make sense for server or long-lived process, but not best for development or CI cost which people often neglected), but by total performance which consist of Compile duration (if any) + VM startup duration (if any) + AOT or JIT duration (if any) + Runtime duration, so every strategy the PL's implementator use can be fairly judged.

3. Why there's no C++, VB.NET, C#, D, Object-Pascal? don't want to compile things (since there's no build and run command in one flag).  

4. Why there's no Kotlin, Scala, Rust, Elixir, Pony, Swift, Groovy, or Zig? Too lazy to add :3 you can contribute tho (create a pull request, then I'll run the benchmark again as preferably as there's precompiled binary/deb/apt/ppa repository for the compiler/interpreter).

5. Why there's no Ruby 3.1? I can't find any PPA for latest Ruby, latest one on Ubuntu 21.10 repo is Ruby2.7.

Contributors: ilmanzo (Nim, Crystal, D), inkydragon (Julia)

Prime Benchmark

So, yesterday I didn't touch my PC to do a prime number benchmark. And Here's the result for single threaded (only showing fastest implementation of each language) and multithreaded:

Index	Implementation	S	Label	Passes	Dur	Algo	Faithful	Bit	Passes/Second
1	cpp	3	flo80_	220667	5.00	base	no	1	44133.26760
10	c	2	daniel	20250	5.00	wheel	yes	1	4049.85259
11	zig	3	ManDeJ	17964	5.00	base	no	1	3592.49823
12	c	2	daniel	17681	5.00	wheel	yes	1	3536.07129
16	rust	1	mike-b	15804	5.01	base	yes	8	3152.68929
20	assembly	1	rberge	14434	5.00	base	no	8	2886.80000
22	haskell	1	fatho/	11959	5.00	base	no	8	2391.77321
32	fortran	1	johand	9987	5.00	base	no	1	1997.40000
36	crystal	1	marghi	8680	5.00	base	yes	1	1735.86981
38	fsharp	3	dmanno	7754	5.00	base	yes		1550.68897
40	java	1	Mansen	14887	10.0	base	yes		1488.70000
41	csharp	1	kinema	7271	5.00	base	yes		1454.08077
43	julia	2	epithe	6953	5.00	base	yes	1	1390.55577
46	go	2	ssoves	6161	5.00	base	yes	1	1232.01471
51	nodejs	1	rogier	5748	5.00	base	yes	1	1149.43213
57	lisp	2	mayerr	5122	5.00	base	no	1	1024.19803
58	typescript	1	marghi	5031	5.00	base	yes		1006.20000
59	d	2	Bradle	5003	5.00	base	yes	1	1000.52396
61	v	1	marghi	4329	5.00	base	yes		865.80000
63	lua	2	ben1je	3159	5.00	base	no	1	631.80000
64	nim	2	beef33	2871	5.00	base	yes	1	574.02096
67	cython	1	rpkak	2659	5.00	base	yes	8	531.64832
71	basic	1	rberge	2416	5.00	wheel	yes	1	483.00680
73	assemblyscript	1	donmah	4231	10.0	base	yes		423.05768
74	python	2	ssoves	1991	5.00	base	yes	8	398.09742
80	scala	1	rom1de	1203	5.00	base	yes		240.55189
81	pascal	1	rberge	1162	5.00	base	yes		232.40000
82	cobol	1	fvbake	1157	5.00	base	no	8	231.40000
83	pony	1	marghi	1144	5.00	base	yes	1	228.80000
84	swift	1	j-f1	2046	10.0	base	yes		204.55332
85	dart	1	eagere	824	5.00	base	yes		164.77795
86	haxe	1	TayIor	1392	10.0	base	yes		139.19035
88	ada	1	BoopBe	661	5.00	base	no		132.02220
92	octave	1	octave	313	5.00	base	no		62.54234
93	postscript	1	epithe	216	5.01	base	no	8	43.08797
94	ruby	1	rberge	119	5.01	base	yes		23.71935
95	wren	1	marghi	111	5.00	base	yes		22.16446
96	php	1	Dennis	143	10.0	base	yes		14.24667
97	smalltalk	1	fvbake	49	5.07	base	yes	1	9.66469
99	mixal	1	rberge	30	4.91	base	no	1	6.10998
100	perl	1	marghi	28	5.16	base	yes		5.42031
103	r	1	fvbake	7	5.43	base	yes	32	1.28842
104	sql	2	fvbake	6	5.43	other	no	32	1.10375
105	tcl	1	fvbake	6	5.47	base	yes	1	1.09589
111	latex	1	tjol	2	17.8	base	no	32	0.11224
112	bash	1	bash	1	10.6	base	no		0.09357

Index	Implementation	S	Label	Passes	Dur	Thread	Algo	Faithful	Bit	Passes/Second
1	zig	3	ManDe	140910	5.0	4	wheel	no	1	7045.26046
2	cpp	3	flo80	236184	5.0	8	base	no	1	5904.57992
3	zig	3	ManDe	106399	5.0	4	wheel	no	1	5319.64146
4	zig	3	ManDe	101026	5.0	4	wheel	no	1	5051.08785
5	zig	3	ManDe	84002	5.0	4	wheel	no	1	4200.08320
6	zig	3	ManDe	147822	5.0	8	wheel	no	1	3695.38740
7	zig	3	ManDe	72021	5.0	4	wheel	no	1	3600.73314
8	zig	3	ManDe	70522	5.0	4	wheel	no	1	3525.83204
9	zig	3	ManDe	134124	5.0	8	wheel	no	1	3352.91894
10	zig	3	ManDe	59851	5.0	4	base	no	8	2992.45424
11	c	2	danie	101391	5.0	8	wheel	yes	1	2534.43285
12	zig	3	ManDe	98246	5.0	8	wheel	no	1	2455.98299
13	zig	3	ManDe	98190	5.0	8	wheel	no	1	2454.57327
14	zig	3	ManDe	48164	5.0	4	base	no	1	2408.10849
15	zig	3	ManDe	91745	5.0	8	wheel	no	1	2293.50574
16	zig	3	ManDe	90598	5.0	8	wheel	no	1	2264.84129
17	c	2	danie	88103	5.0	8	wheel	yes	1	2199.97727
18	zig	3	ManDe	42318	5.0	4	base	no	1	2115.85768
19	c	2	danie	78858	5.0	8	wheel	yes	1	1969.05838
20	zig	3	ManDe	68492	5.0	8	base	no	8	1712.11852
21	c	2	danie	63752	5.0	8	wheel	yes	1	1591.86334
22	rust	1	mike-	59001	5.0	8	base	yes	8	1474.83765
23	rust	1	mike-	52979	5.0	8	base	yes	1	1324.32205
24	c	2	danie	49822	5.0	8	base	yes	1	1244.43126
25	zig	3	ManDe	49712	5.0	8	base	no	1	1242.59124
26	c	2	danie	24664	5.0	4	wheel	yes	1	1233.15067
27	rust	1	mike-	49238	5.0	8	base	yes	1	1230.78145
28	zig	3	ManDe	45636	5.0	8	base	no	1	1140.80189
29	c	2	danie	22378	5.0	4	wheel	yes	1	1118.88478
30	c	2	danie	20385	5.0	4	wheel	yes	1	1019.22065
31	c	2	danie	20257	5.0	4	wheel	yes	1	1012.82346
32	c	2	danie	15132	5.0	4	base	yes	1	756.55294
33	cpp	2	davep	20048	5.0	8	base	yes	1	501.18496
34	d	2	Bradl	19553	5.0	8	base	yes	1	488.74973
35	zig	3	ManDe	9240	5.0	4	base	no	8	461.94272
36	zig	3	ManDe	11834	5.0	8	base	no	8	295.81332
37	csharp	1	kinem	3193	5.0	8	wheel	yes	1	79.81766
38	csharp	1	kinem	2961	5.0	8	base	yes	1	74.02041

Raw result on this gist

String Associative Array and CombSort Benchmark 2020 Edition

5 Years later since last string associative benchmark and lesser string associative benchmark (measuring string concat operation and built-in associative array set and get), numeric comb sort benchmark and string comb sort benchmark (measuring basic array random access, string conversion, and array swap for number and string), this year's using newer processor: AMD Ryzen 3 3100 running on 64-bit Ubuntu 20.04. Now with 10x more data to hopefully make the benchmark runs 10x slower (at least 1 sec), best of 3 runs.

$ alias time='/usr/bin/time -f "\nCPU: %Us\tReal: %es\tRAM: %MKB"'

$ php -v

PHP 7.4.3 (cli) (built: Oct  6 2020 15:47:56) ( NTS )

$ time php assoc.php 

637912 641149 67002

3808703 14182513 2343937

CPU: 1.25s      Real: 1.34s     RAM: 190644KB

$ python3 -V

Python 3.8.5

$ time python3 dictionary.py

637912 641149 67002

3808703 14182513 2343937

CPU: 5.33s      Real: 5.47s     RAM: 314564KB

$ ruby3.0 -v

ruby 3.0.0p0 (2020-12-25 revision 95aff21468) [x86_64-linux-gnu]

$ time ruby3.0 --jit hash.rb 

637912 641149 67002

3808703 14182513 2343937

CPU: 6.50s      Real: 5.94s     RAM: 371832KB

$ go version

go version go1.14.7 linux/amd64

$ time go run map.go

637912 641149 67002

3808703 14182513 2343937

CPU: 1.79s      Real: 1.56s     RAM: 257440KB

$ node -v       

v14.15.2

$ time node object.js

637912 641149 67002

3808703 14182513 2343937

CPU: 2.24s      Real: 2.21s     RAM: 326636KB

$ luajit -v 

LuaJIT 2.1.0-beta3 -- Copyright (C) 2005-2017 Mike Pall. http://luajit.org/

$ time luajit table.lua

637912  641149  67002

3808703 14182513        2343937

CPU: 4.11s      Real: 4.22s     RAM: 250828KB

$ dart --version

Dart SDK version: 2.10.4 (stable) (Unknown timestamp) on "linux_x64"

$ time dart map.dart

637912 641149 67002

3808703 14182513 2343937

CPU: 2.99s      Real: 2.91s     RAM: 385496KB

$ v version

V 0.2 36dcace

$ time v run map.v

637912, 641149, 67002

3808703, 14182513, 2343937

CPU: 4.79s      Real: 5.28s     RAM: 1470668KB

$ tcc -v

tcc version 0.9.27 (x86_64 Linux)

$ time tcc -run uthash.c

637912 641149 67002

3808703 14182513 2343937

Command exited with non-zero status 25

CPU: 2.52s      Real: 2.61s     RAM: 291912KB

$ export GOPHERJS_GOROOT="$(go1.12.16 env GOROOT)"

$ npm install --global source-map-support

$ goperjs version

GopherJS 1.12-3

$ time gopherjs 

637912 641149 67002

3808703 14182513 2343937

CPU: 14.13s     Real: 12.01s    RAM: 597712KB

$ java -version

java version "14.0.2" 2020-07-14

Java(TM) SE Runtime Environment (build 14.0.2+12-46)

Java HotSpot(TM) 64-Bit Server VM (build 14.0.2+12-46, mixed mode, sharing)

$ time java hashmap.java

637912 641149 67002

3808703 14182513 2343937

CPU: 5.18s      Real: 1.63s     RAM: 545412KB

The result shows a huge improvement for PHP since the old 5.4. NodeJS also huge improvement compared to old 0.10. The rest is quite bit the same. Also please keep note that Golang and V includes build/compile time not just run duration, and it seems V performance really bad when it comes to string operations (the compile itself really fast, less than 1s for 36dcace -- using gcc 9.3.0). 

Next we're gonna benchmark comb sort implementation. But this time we use jit version of ruby 2.7, since it's far way faster (19s vs 26s and 58s vs 66s for string benchmark), for ruby 3.0 we always use jit version since it's faster than non-jit. In case for C (TCC) which doesn't have built-in associative array, I used uthash, because it's the most popular. TinyGo does not complete first benchmark after more than 1000s, sometimes segfault. XS Javascript engine failed to give correct result, engine262 also failed to finish within 1000s.

Language	Command Flags	Version	Assoc	RAM	Num Comb	RAM	Str Comb	RAM	Total	RAM
Go	go run	1.14.7	1.56	257,440	0.73	82,844	4.74	245,432	7.03	585,716
Go	go run	1.15.6	1.73	256,620	0.78	82,896	4.86	245,468	7.37	584,984
Nim	nim r -d:release --gc:arc	1.4.2	1.56	265,172	0.79	79,284	5.77	633,676	8.12	978,132
Nim	nim r -d:release --gc:orc	1.4.2	1.53	265,160	0.94	79,380	5.83	633,636	8.30	978,176
Javascript	node	14.15.2	2.21	327,048	0.87	111,972	6.13	351,520	9.21	790,540
Crystal	crystal run --release	0.35.1	1.81	283,648	1.44	146,700	6.09	440,796	9.34	871,144
Javascript	~/.esvu/bin/v8	8.9.201	1.77	177,748	0.89	105,416	6.71	335,236	9.37	618,400
C	tcc -run	0.9.27	2.61	291,912	1.45	80,832	6.40	393,352	10.46	766,096
Java	java	14.0.2 2020-07-14	1.63	545,412	1.50	165,864	7.69	743,572	10.82	1,454,848
Nim	nim r -d:release	1.4.2	1.91	247,456	0.96	79,476	8.38	1,211,116	11.25	1,538,048
Dart	dart	2.10.4	2.91	385,496	1.61	191,916	7.31	616,716	11.83	1,194,128
Python	pypy	7.3.1+dfsg-2	2.19	331,776	2.83	139,740	8.04	522,648	13.06	994,164
Javascript	~/.esvu/bin/chakra	1.11.24.0	2.73	487,400	1.27	102,192	11.27	803,168	15.27	1,392,760
Javascript	~/.esvu/bin/jsc	271117	5.90	593,624	0.68	111,972	9.09	596,088	15.67	1,301,684
V	v -prod run	0.2 32091dd gcc-10.2	4.78	1,469,932	1.86	79,376	14.06	1,560,516	20.70	3,109,824
Lua	luajit	2.1.0-beta3	4.11	250,828	3.76	133,424	12.91	511,196	20.78	895,448
Javascript	~/.esvu/bin/sm	JavaScript-C86.0a1	5.61	378,064	1.40	96,480	13.81	393,376	20.82	867,920
V	v -prod run	0.2 32091dd gcc-9.3	5.05	1,469,936	2.14	79,408	14.62	1,560,484	21.81	3,109,828
Javascript	~/.esvu/bin/graaljs	CE Native 20.3.0	7.78	958,380	4.45	405,900	14.31	911,220	26.54	2,275,500
Go	gopherjs run	1.12-3 (node 14.15.2)	11.76	594,896	2.04	119,604	18.46	397,396	32.26	1,111,896
Nim	nim r	1.4.2	6.60	247,444	3.05	79,332	31.85	1,211,208	41.50	1,537,984
PHP	php	7.4.3	1.34	190,644	10.11	328,452	34.51	641,664	45.96	1,160,760
Ruby	truffleruby	21.1.0-dev-c1517c55	14.54	2,456,156	3.09	453,152	29.27	3,660,284	46.90	6,569,592
Crystal	crystal run	0.35.1	5.69	284,328	12.00	153,828	31.69	441,740	49.38	879,896
Javascript	~/.esvu/bin/quickjs	2020-11-08	3.90	252,484	23.48	80,772	34.80	471,624	62.18	804,880
V	v run	0.2 36dcace gcc-9.3	5.28	1,470,668	6.60	80,232	58.99	1,561,176	70.87	3,112,076
Lua	lua	5.3.3	5.98	366,516	27.26	264,648	46.05	864,300	79.29	1,495,464
Ruby	ruby	2.7.0p0	6.31	371,456	19.29	100,536	58.82	694,560	84.42	1,166,552
Python	python3	3.8.5	5.47	314,564	33.96	404,976	47.79	722,820	87.22	1,442,360
Ruby	jruby	9.2.9.0	7.45	1,878,184	34.11	1,976,844	59.83	7,115,448	101.39	10,970,476
Ruby	ruby	3.0.0p0	5.94	371,832	24.87	92,844	74.32	1,015,096	105.13	1,479,772
Go	tinygo run	0.16.0	999.99	318,148	3.68	300,548	252.34	711,340	1256.01	1,330,036

Golang still the winner (obviously, since it's compiled), then Nim (Compiled), next best JIT or interpreter is NodeJS, Crystal (Compiled, not JIT), v8, followed Java, by TCC (Compiled) Dart, PyPy, V (Compiled, not JIT), LuaJIT, PHP, Ruby, and Python3. The recap spreadsheet can be accessed here. 

FAQ:

1. Why you measure the compile duration too? because developer experience also important (feedback loop), at least for me.

2. Why not warming up the VM first? each implementation have it's own advantage and disadvantage.

3. Why there's no C++, VB.NET, C#, D, Object-Pascal? don't want to compile things (since there's no build and run command in one flag).  

4. Why there's no Kotlin, Scala, Rust, Pony, Swift, Groovy, Julia, Crystal, or Zig? Too lazy to add :3 you can contribute tho (create a pull request, then I'll run the benchmark again as preferabbly as there's precompiled binary/deb/apt/ppa repository for the compiler/interpreter).

Contributors: ilmanzo (Nim, Crystal, D)