2015-02-17

String Associative Array Benchmark

Today we will benchmark about associative array. Most associative array are implemented using hashtable or tree. We'll benchmark all built-in associative array for storing strings, that are C++'s map and unordered_map, Java's TreeMap and HashMap, PHP's Array, Ruby's Hash, JavaScript's Object, Go's map, Python's Dictionary, and C#'s Dictionary. The benchmark should only use built-in associative array, integer conversion, and string reverse function. The key to be tested are number-string, inverted, and reversed hex number from N to 0. The value to be stored are double precision floating point (if possible). The key must be searched first, when the key exists, the value incremented by the certain value. After building the associative array, iterate one by one and get the total of first digit of each value, total length of the key, and total number of entries. We also want to compare the performance some of the best data structures (in my experience, in terms of memory usage and speed ratio), that is 256-ary radix tree (judy-template), double-array trie (cedar), and HAT-trie (hat-trie). You can look up the source on my dropbox (folder: assoc), sorry, too lazy to use github ;3 This benchmark performed on 64-bit Linux, AMD A8-6600K, 16GB RAM, Non-SSD harddisk.

$ alias | grep 'alias time'
alias time='/usr/bin/time -f "\nCPU: %Us\tReal: %es\tRAM: %MKB"'
$ time --version
GNU time 1.7

$ g++ --version
g++ (GCC) 4.9.2 20141224 (prerelease)

$ time g++ -std=c++11 map.cpp
CPU: 0.33s      Real: 0.39s     RAM: 57764KB
$ time ./a.out
6009354 6009348 611297
36186112 159701682 23370001
CPU: 141.03s    Real: 142.40s   RAM: 2558828KB

$ time g++ -std=c++11 -O2 map.cpp
CPU: 0.40s      Real: 0.45s     RAM: 59672KB
$ time ./a.out
6009354 6009348 611297
36186112 159701682 23370001
CPU: 65.37s     Real: 66.63s    RAM: 2558720KB

$ time g++ -std=c++11 unordered_map.cpp
CPU: 0.37s      Real: 0.44s     RAM: 61804KB
$ time ./a.out
6009354 6009348 611297
36186112 159701682 23370001
CPU: 45.46s     Real: 46.68s    RAM: 2478132KB

$ time g++ -std=c++11 -O2 unordered_map.cpp
CPU: 0.41s      Real: 0.45s     RAM: 62688KB
$ time ./a.out
6009354 6009348 611297
36186112 159701682 23370001
CPU: 29.16s     Real: 30.32s    RAM: 2478148KB

$ time g++ -std=c++11 judy.cpp
CPU: 0.29s      Real: 0.34s     RAM: 50088KB
$ time ./a.out
6009354 6009348 611297
36186112 159701681 23370000
CPU: 27.07s     Real: 27.46s    RAM: 601604KB

$ time g++ -std=c++11 -O2 judy.cpp
CPU: 0.57s      Real: 0.62s     RAM: 55192KB
$ time ./a.out
6009354 6009348 611297
36186112 159701681 23370000
CPU: 16.31s     Real: 16.62s    RAM: 601600KB

$ time g++ -std=c++11 cedar.cpp
CPU: 0.36s      Real: 0.40s     RAM: 55344KB
$ time ./a.out
6009354 6009348 611297
36186112 159701682 23370001
CPU: 37.97s     Real: 38.23s    RAM: 749144KB

$ time g++ -std=c++11 -O2 cedar.cpp
CPU: 0.93s      Real: 0.97s     RAM: 70436KB
$ time ./a.out
6009354 6009348 611297
36186112 159701682 23370001
CPU: 20.61s     Real: 20.89s    RAM: 746352KB

$ time g++ -std=c++11 hat_trie.cpp
CPU: 0.31s      Real: 0.36s     RAM: 49844KB
$ time ./a.out
6009354 6009348 611297
36186112 159701682 23370001
CPU: 41.99s     Real: 42.42s    RAM: 567048KB

$ time g++ -std=c++11 -O2 hat_trie.cpp
CPU: 0.54s      Real: 0.59s     RAM: 55376KB
$ time ./a.out
6009354 6009348 611297
36186112 159701682 23370001
CPU: 27.03s     Real: 27.37s    RAM: 567040KB

$ clang --version
clang version 3.5.1 (tags/RELEASE_351/final)

$ time clang++ -std=c++11 map.cpp
CPU: 0.30s      Real: 0.33s     RAM: 49828KB
$ time ./a.out
6009354 6009348 611297
36186112 159701682 23370001
CPU: 138.63s    Real: 140.16s   RAM: 2558732KB

$ time clang++ -std=c++11 -O2 map.cpp
CPU: 0.39s      Real: 0.42s     RAM: 53680KB
$ time ./a.out
6009354 6009348 611297
36186112 159701682 23370001
CPU: 67.43s     Real: 68.73s    RAM: 2558720KB

$ time clang++ -std=c++11 unordered_map.cpp
CPU: 0.32s      Real: 0.36s     RAM: 52148KB
$ time ./a.out
6009354 6009348 611297
36186112 159701682 23370001
CPU: 45.22s     Real: 46.46s    RAM: 2478136KB

$ time clang++ -std=c++11 -O2 unordered_map.cpp
CPU: 0.41s      Real: 0.45s     RAM: 54828KB
$ time ./a.out
6009354 6009348 611297
36186112 159701682 23370001
CPU: 30.13s     Real: 31.30s    RAM: 2478120KB

$ time clang++ -std=c++11 -w judy.cpp
CPU: 0.26s      Real: 0.29s     RAM: 46872KB
$ time ./a.out
6009354 6009348 611297
36186112 159701681 23370000
CPU: 27.88s     Real: 28.21s    RAM: 601636KB

$ time clang++ -std=c++11 -O2 -w judy.cpp
CPU: 0.46s      Real: 0.50s     RAM: 51460KB
$ time ./a.out
6009354 6009348 611297
36186112 159701681 23370000
CPU: 17.27s     Real: 17.64s    RAM: 601600KB

$ time clang++ -std=c++11 cedar.cpp
CPU: 0.30s      Real: 0.35s     RAM: 50012KB
$ time ./a.out
6009354 6009348 611297
36186112 159701682 23370001
CPU: 38.26s     Real: 38.54s    RAM: 748868KB

$ time clang++ -std=c++11 -O2 cedar.cpp
CPU: 0.48s      Real: 0.51s     RAM: 53152KB
$ time ./a.out
6009354 6009348 611297
36186112 159701682 23370001
CPU: 21.95s     Real: 22.19s    RAM: 749248KB

$ time clang++ -std=c++11 hat_trie.cpp
CPU: 0.26s      Real: 0.29s     RAM: 47700KB
$ time ./a.out
6009354 6009348 611297
36186112 159701682 23370001
CPU: 44.64s     Real: 45.04s    RAM: 567016KB

$ time clang++ -std=c++11 -O2 hat_trie.cpp
CPU: 0.50s      Real: 0.54s     RAM: 52532KB
$ time ./a.out
6009354 6009348 611297
36186112 159701682 23370001
CPU: 27.24s     Real: 27.57s    RAM: 567040KB

$ javac -version
javac 1.7.0_75

$ time javac tree_map.java
CPU: 1.28s      Real: 0.86s     RAM: 62040KB
$ time java tree_map
6009354 6009348 611297
36186112 159701682 23370001
CPU: 233.50s    Real: 101.05s   RAM: 3951752KB

$ time javac hash_map.java
CPU: 1.21s      Real: 0.84s     RAM: 63588KB
$ time java hash_map
6009354 6009348 611297
36186112 159701682 23370001
CPU: 345.03s    Real: 103.54s   RAM: 4119388KB

$ ruby --version
ruby 2.2.0p0 (2014-12-25 revision 49005) [x86_64-linux]
$ time ruby hash.rb
6009354 6009348 611297
36186112 159701682 23370001
CPU: 107.91s    Real: 109.66s   RAM: 3031872KB

$ jruby -version
jruby 9.0.0.0.pre1 (2.2.0p0) 2015-01-20 d537cab OpenJDK 64-Bit Server VM 24.75-b04 on 1.7.0_75-b13 +jit [linux-amd64]
$ time jruby -J-Xmx12000M -J-Djruby.compile.mode=FORCE hash.rb
6009354 6009348 611297
36186112 159701682 23370001
CPU: 421.81s    Real: 180.57s   RAM: 8740612KB

$ go version go version go1.4.1 linux/amd64 $ time go build map.go
CPU: 0.15s      Real: 0.20s     RAM: 34892KB
$ time ./map
6009354 6009348 611297
36186112 159701682 23370001
CPU: 24.10s     Real: 25.01s    RAM: 2748784KB

$ pypy --version
Python 2.7.8 (c6ad44ecf5d8, Nov 18 2014, 18:04:31) [PyPy 2.4.0 with GCC 4.9.2]
$ time pypy dictionary.py
(6009354, 6009348, 611297)
(36186112, 159701682, 23370001)
CPU: 79.87s     Real: 84.57s    RAM: 4034956KB

$ mcs --version
Mono C# compiler version 3.12.0.0
$ time mcs dictionary.cs
CPU: 0.48s      Real: 0.51s     RAM: 48188KB
$ time mono ./dictionary.exe
6009354 6009348 611297
36186112 159701682 23370001
CPU: 40.98s     Real: 42.07s    RAM: 1690224KB

Edit, after bugfixing the code for PHP, Python and JavaScript source, only JavaScriptCore, Rhino and Python3 that successfully complete the benchmark.

$ pacman -Qo `which jsc-3`
/usr/bin/jsc-3 is owned by webkitgtk 2.4.8-1
$ time jsc-3 object.js
6009354 6009348 611297
36186112 159701682 23370001
CPU: 148.09s    Real: 121.65s   RAM: 4666280KB

$ rhino < /dev/null
Rhino 1.7 release 4 2014 07 01
$ time rhino object.js
6009354 6009348 611297
36186112 159701682 23370001
CPU: 524.12s    Real: 189.04s   RAM: 4220872KB

$ python3 --version
Python 3.4.2
$ time python3 dictionary.py
6009354 6009348 611297
36186112 159701682 23370001
CPU: 157.71s    Real: 163.37s   RAM: 4335568KB

$ lua -v
Lua 5.2.3  Copyright (C) 1994-2013 Lua.org, PUC-Rio
$ time lua table.lua
6009354 6009348 611297
36186112        159701682       23370001
CPU: 101.73s    Real: 106.45s   RAM: 3079336KB

$ dart --version
Dart VM version: 1.8.5 (Tue Jan 13 12:44:14 2015) on "linux_x64"
$ time dart --old_gen_heap_size=4096 map.dart
6009354 6009348 611297
36186112 159701682 23370001
CPU: 133.35s    Real: 133.64s   RAM: 2538220KB

And the summary

Compiler / InterpreterLanguageType NameData StructureCompile DurationCompile RAMRuntime DurationRuntime RAMCorrect?
g++ (debug)C++mapRB-Tree330577641410302558828
g++ (-O2)C++mapRB-Tree40059672653702558720
g++ (debug)C++unordered_mapHash Table37061804454602478132
g++ (-O2)C++unordered_mapHash Table41062688291602478148
g++ (debug)C++judySArray256-ary Radix Tree2905008827070601604-2
g++ (-O2)C++judySArray256-ary Radix Tree5705519216310601600-2
g++ (debug)C++cedar::daDouble Array Trie3605534437970749144
g++ (-O2)C++cedar::daDouble Array Trie9307043620610746352
g++ (debug)C++hattrie_tHAT-Trie3104984441990567048
g++ (-O2)C++hattrie_tHAT-Trie5405537627030567040
clang++ (debug)C++mapRB-Tree300498281386302558732
clang++ (-O2)C++mapRB-Tree39053680674302558720
clang++ (debug)C++unordered_mapHash Table32052148452002478136
clang++ (-O2)C++unordered_mapHash Table41054828301302478120
clang++ (debug)C++judySArray256-ary Radix Tree2604687227880601636-2
clang++ (-O2)C++judySArray256-ary Radix Tree4605146017270601600-2
clang++ (debug)C++cedar::daDouble Array Trie3005001238260748868
clang++ (-O2)C++cedar::daDouble Array Trie4805316021950749248
clang++ (debug)C++hattrie_tHAT-Trie2604790844640567016
clang++ (-O2)C++hattrie_tHAT-Trie5005253227240567040
javac, javaJavaTreeMapRB-Tree1280620401010503951752
javac, javaJavaHashMapHash Table1210635881035404119388
rubyRubyHashHash Table1079103031872
jrubyRubyHashHash Table1805708740612
jsc-3JavaScriptObjectUnknown1216504666280
rhinoJavaScriptObjectUnknown1890404220872
goGomapHash Table15034892241002748784
python3Python 3dictHash Table1577104335568
pypyPython 2dictHash Table798704034956
mcsC#DictionaryHash Table48048188409801690224
luaLuatableHash Table1017303079336
dartDartMapHash Table1333502538220

Note #1: PHP 5.6.5, HHVM 3.5.0, Rubinius 2.5.2, NodeJS 0.10.35 failed to build the associative array within 300s.

Note #2: again MozJS 2.4 got uncaught exception: out of memory when it uses 1.2GB of RAM

Note #3: LuaJIT 2.0.3 got not enough memory when it uses 1GB of RAM

No comments :

Post a Comment

THINK: is it True? is it Helpful? is it Inspiring? is it Necessary? is it Kind?