Just want to share some table from initial chapter of my thesis (early 2012), it's about modification (added lots of compression) of HAT-Trie for DNS suffix blocking. These tables are from chapter IV, since it's the only exciting part about it .__.)/||. That time I didn't know about Cedar yet (of course it's 2013 XD). This is the list of benchmarked data structure:
The data structure name that marked with "*" sign are those who can be set as nested per subdomain (that is should be a map/associative array). The strings tested are C++'s std::string, Qt's QString, csubdom (compressed subdomain), strcsubdom (compressed subdomain, stored in std::string), clabels (compressed full domain), strclables (compressed full domain name). Compressed in this term are packed characters (from 8-bit to 5/6-bit so it would use less memory). This benchmark performed on Ubuntu 64-bit Linux 3.2, GCC 4.6.3, AMD Phenom X9850, 8GB RAM and non-SSD disk, compile flag: g++ -c -m64 -pipe -O2 -Wall -W.
Notes about that table header "insert" is a benchmark about inserting 2.1 million blacklisted domain names, after it's completed, the data structure erased and the insert operation repeated until 30 seconds passes. The "misses" benchmark about checking if 68.4k domains that doesn't exists on the blacklist, the operation repeated until 2 seconds passes. The "exists" benchmark is about rechecking blacklisted domain names in sorted order, repeated until 8 seconds passes. The "random" benchmark is about checking random blacklisted items, repeated until 20 seconds passes. The value there are number of milliseconds required per domain name. Last column on the table is average bytes required to store one domain name.
programming: the action or process of writing computer programs. | rants: speak or shout at length in a wild, [im]passioned way.
Showing posts with label hat-trie. Show all posts
Showing posts with label hat-trie. Show all posts
2015-02-27
2015-02-17
String Associative Array Benchmark
Today we will benchmark about associative array. Most associative array are implemented using hashtable or tree. We'll benchmark all built-in associative array for storing strings, that are C++'s map and unordered_map, Java's TreeMap and HashMap, PHP's Array, Ruby's Hash, JavaScript's Object, Go's map, Python's Dictionary, and C#'s Dictionary. The benchmark should only use built-in associative array, integer conversion, and string reverse function. The key to be tested are number-string, inverted, and reversed hex number from N to 0. The value to be stored are double precision floating point (if possible). The key must be searched first, when the key exists, the value incremented by the certain value. After building the associative array, iterate one by one and get the total of first digit of each value, total length of the key, and total number of entries. We also want to compare the performance some of the best data structures (in my experience, in terms of memory usage and speed ratio), that is 256-ary radix tree (judy-template), double-array trie (cedar), and HAT-trie (hat-trie). You can look up the source on my dropbox (folder: assoc), sorry, too lazy to use github ;3 This benchmark performed on 64-bit Linux, AMD A8-6600K, 16GB RAM, Non-SSD harddisk.
$ alias | grep 'alias time'
alias time='/usr/bin/time -f "\nCPU: %Us\tReal: %es\tRAM: %MKB"'
$ time --version
GNU time 1.7
$ g++ --version
g++ (GCC) 4.9.2 20141224 (prerelease)
$ time g++ -std=c++11 map.cpp
CPU: 0.33s Real: 0.39s RAM: 57764KB
$ time ./a.out
6009354 6009348 611297
36186112 159701682 23370001
CPU: 141.03s Real: 142.40s RAM: 2558828KB
$ time g++ -std=c++11 -O2 map.cpp
CPU: 0.40s Real: 0.45s RAM: 59672KB
$ time ./a.out
6009354 6009348 611297
36186112 159701682 23370001
$ alias | grep 'alias time'
alias time='/usr/bin/time -f "\nCPU: %Us\tReal: %es\tRAM: %MKB"'
$ time --version
GNU time 1.7
$ g++ --version
g++ (GCC) 4.9.2 20141224 (prerelease)
$ time g++ -std=c++11 map.cpp
CPU: 0.33s Real: 0.39s RAM: 57764KB
$ time ./a.out
6009354 6009348 611297
36186112 159701682 23370001
CPU: 141.03s Real: 142.40s RAM: 2558828KB
$ time g++ -std=c++11 -O2 map.cpp
CPU: 0.40s Real: 0.45s RAM: 59672KB
$ time ./a.out
6009354 6009348 611297
36186112 159701682 23370001
CPU: 65.37s Real: 66.63s RAM: 2558720KB
$ time g++ -std=c++11 unordered_map.cpp
CPU: 0.37s Real: 0.44s RAM: 61804KB
$ time ./a.out
6009354 6009348 611297
36186112 159701682 23370001
$ time g++ -std=c++11 unordered_map.cpp
CPU: 0.37s Real: 0.44s RAM: 61804KB
$ time ./a.out
6009354 6009348 611297
36186112 159701682 23370001
CPU: 45.46s Real: 46.68s RAM: 2478132KB
$ time g++ -std=c++11 -O2 unordered_map.cpp
CPU: 0.41s Real: 0.45s RAM: 62688KB
$ time ./a.out
6009354 6009348 611297
36186112 159701682 23370001
$ time g++ -std=c++11 -O2 unordered_map.cpp
CPU: 0.41s Real: 0.45s RAM: 62688KB
$ time ./a.out
6009354 6009348 611297
36186112 159701682 23370001
CPU: 29.16s Real: 30.32s RAM: 2478148KB
$ time g++ -std=c++11 judy.cpp
CPU: 0.29s Real: 0.34s RAM: 50088KB
$ time ./a.out
6009354 6009348 611297
36186112 159701681 23370000
CPU: 27.07s Real: 27.46s RAM: 601604KB
$ time g++ -std=c++11 -O2 judy.cpp
CPU: 0.57s Real: 0.62s RAM: 55192KB
$ time ./a.out
6009354 6009348 611297
36186112 159701681 23370000
$ time g++ -std=c++11 judy.cpp
CPU: 0.29s Real: 0.34s RAM: 50088KB
$ time ./a.out
6009354 6009348 611297
36186112 159701681 23370000
CPU: 27.07s Real: 27.46s RAM: 601604KB
$ time g++ -std=c++11 -O2 judy.cpp
CPU: 0.57s Real: 0.62s RAM: 55192KB
$ time ./a.out
6009354 6009348 611297
36186112 159701681 23370000
CPU: 16.31s Real: 16.62s RAM: 601600KB
$ time g++ -std=c++11 cedar.cpp
CPU: 0.36s Real: 0.40s RAM: 55344KB
$ time ./a.out
6009354 6009348 611297
36186112 159701682 23370001
$ time g++ -std=c++11 -O2 cedar.cpp
CPU: 0.93s Real: 0.97s RAM: 70436KB
$ time g++ -std=c++11 hat_trie.cpp
$ time g++ -std=c++11 -O2 hat_trie.cpp
$ clang --version
$ time clang++ -std=c++11 map.cpp
$ time clang++ -std=c++11 unordered_map.cpp
CPU: 0.32s Real: 0.36s RAM: 52148KB
$ time clang++ -std=c++11 -O2 unordered_map.cpp
CPU: 0.41s Real: 0.45s RAM: 54828KB
$ time clang++ -std=c++11 -w judy.cpp
CPU: 0.26s Real: 0.29s RAM: 46872KB
$ time clang++ -std=c++11 -O2 -w judy.cpp
CPU: 0.46s Real: 0.50s RAM: 51460KB
$ time clang++ -std=c++11 cedar.cpp
CPU: 0.30s Real: 0.35s RAM: 50012KB
$ time clang++ -std=c++11 -O2 cedar.cpp
CPU: 0.48s Real: 0.51s RAM: 53152KB
$ time clang++ -std=c++11 hat_trie.cpp
$ time clang++ -std=c++11 -O2 hat_trie.cpp
$ javac -version
$ time javac tree_map.java
$ time javac hash_map.java
$ jruby -version
Note #1: PHP 5.6.5, HHVM 3.5.0, Rubinius 2.5.2, NodeJS 0.10.35 failed to build the associative array within 300s.
Note #2: again MozJS 2.4 got uncaught exception: out of memory when it uses 1.2GB of RAM
Note #3: LuaJIT 2.0.3 got not enough memory when it uses 1GB of RAM
$ time g++ -std=c++11 cedar.cpp
CPU: 0.36s Real: 0.40s RAM: 55344KB
$ time ./a.out
6009354 6009348 611297
36186112 159701682 23370001
CPU: 37.97s Real: 38.23s RAM: 749144KB
$ time g++ -std=c++11 -O2 cedar.cpp
CPU: 0.93s Real: 0.97s RAM: 70436KB
$ time ./a.out
6009354 6009348 611297
6009354 6009348 611297
36186112 159701682 23370001
CPU: 20.61s Real: 20.89s RAM: 746352KB
$ time g++ -std=c++11 hat_trie.cpp
CPU: 0.31s Real: 0.36s RAM: 49844KB
$ time ./a.out
6009354 6009348 611297
6009354 6009348 611297
36186112 159701682 23370001
CPU: 41.99s Real: 42.42s RAM: 567048KB
$ time g++ -std=c++11 -O2 hat_trie.cpp
CPU: 0.54s Real: 0.59s RAM: 55376KB
$ time ./a.out
6009354 6009348 611297
6009354 6009348 611297
36186112 159701682 23370001
CPU: 27.03s Real: 27.37s RAM: 567040KB
$ clang --version
clang version 3.5.1 (tags/RELEASE_351/final)
$ time clang++ -std=c++11 map.cpp
CPU: 0.30s Real: 0.33s RAM: 49828KB
$ time ./a.out
6009354 6009348 611297
6009354 6009348 611297
36186112 159701682 23370001
CPU: 138.63s Real: 140.16s RAM: 2558732KB
$ time clang++ -std=c++11 -O2 map.cpp
CPU: 0.39s Real: 0.42s RAM: 53680KB
$ time ./a.out
6009354 6009348 611297
6009354 6009348 611297
36186112 159701682 23370001
CPU: 67.43s Real: 68.73s RAM: 2558720KB
$ time clang++ -std=c++11 unordered_map.cpp
CPU: 0.32s Real: 0.36s RAM: 52148KB
$ time ./a.out
6009354 6009348 611297
6009354 6009348 611297
36186112 159701682 23370001
CPU: 45.22s Real: 46.46s RAM: 2478136KB
$ time clang++ -std=c++11 -O2 unordered_map.cpp
CPU: 0.41s Real: 0.45s RAM: 54828KB
$ time ./a.out
6009354 6009348 611297
6009354 6009348 611297
36186112 159701682 23370001
CPU: 30.13s Real: 31.30s RAM: 2478120KB
$ time clang++ -std=c++11 -w judy.cpp
CPU: 0.26s Real: 0.29s RAM: 46872KB
$ time ./a.out
6009354 6009348 611297
6009354 6009348 611297
36186112 159701681 23370000
CPU: 27.88s Real: 28.21s RAM: 601636KB
$ time clang++ -std=c++11 -O2 -w judy.cpp
CPU: 0.46s Real: 0.50s RAM: 51460KB
$ time ./a.out
6009354 6009348 611297
6009354 6009348 611297
36186112 159701681 23370000
CPU: 17.27s Real: 17.64s RAM: 601600KB
$ time clang++ -std=c++11 cedar.cpp
CPU: 0.30s Real: 0.35s RAM: 50012KB
$ time ./a.out
6009354 6009348 611297
6009354 6009348 611297
36186112 159701682 23370001
CPU: 38.26s Real: 38.54s RAM: 748868KB
$ time clang++ -std=c++11 -O2 cedar.cpp
CPU: 0.48s Real: 0.51s RAM: 53152KB
$ time ./a.out
6009354 6009348 611297
6009354 6009348 611297
36186112 159701682 23370001
CPU: 21.95s Real: 22.19s RAM: 749248KB
$ time clang++ -std=c++11 hat_trie.cpp
CPU: 0.26s Real: 0.29s RAM: 47700KB
$ time ./a.out
6009354 6009348 611297
6009354 6009348 611297
36186112 159701682 23370001
CPU: 44.64s Real: 45.04s RAM: 567016KB
$ time clang++ -std=c++11 -O2 hat_trie.cpp
CPU: 0.50s Real: 0.54s RAM: 52532KB
$ time ./a.out
6009354 6009348 611297
6009354 6009348 611297
36186112 159701682 23370001
CPU: 27.24s Real: 27.57s RAM: 567040KB
$ javac -version
javac 1.7.0_75
$ time javac tree_map.java
CPU: 1.28s Real: 0.86s RAM: 62040KB
$ time java tree_map
6009354 6009348 611297
36186112 159701682 23370001
36186112 159701682 23370001
CPU: 233.50s Real: 101.05s RAM: 3951752KB
$ time javac hash_map.java
CPU: 1.21s Real: 0.84s RAM: 63588KB
$ time java hash_map
6009354 6009348 611297
36186112 159701682 23370001
CPU: 345.03s Real: 103.54s RAM: 4119388KB
$ ruby --version
ruby 2.2.0p0 (2014-12-25 revision 49005) [x86_64-linux]
ruby 2.2.0p0 (2014-12-25 revision 49005) [x86_64-linux]
$ time ruby hash.rb
6009354 6009348 611297
36186112 159701682 23370001
CPU: 107.91s Real: 109.66s RAM: 3031872KB
$ jruby -version
jruby 9.0.0.0.pre1 (2.2.0p0) 2015-01-20 d537cab OpenJDK 64-Bit Server VM 24.75-b04 on 1.7.0_75-b13 +jit [linux-amd64]
$ time jruby -J-Xmx12000M -J-Djruby.compile.mode=FORCE hash.rb
6009354 6009348 611297
36186112 159701682 23370001
CPU: 421.81s Real: 180.57s RAM: 8740612KB
CPU: 0.15s Real: 0.20s RAM: 34892KB
$ time ./map
6009354 6009348 611297
36186112 159701682 23370001
CPU: 24.10s Real: 25.01s RAM: 2748784KB
$ pypy --version
Python 2.7.8 (c6ad44ecf5d8, Nov 18 2014, 18:04:31) [PyPy 2.4.0 with GCC 4.9.2]
$ time pypy dictionary.py
(6009354, 6009348, 611297)
(36186112, 159701682, 23370001)
CPU: 79.87s Real: 84.57s RAM: 4034956KB
(36186112, 159701682, 23370001)
CPU: 79.87s Real: 84.57s RAM: 4034956KB
$ mcs --version
Mono C# compiler version 3.12.0.0
$ time mcs dictionary.cs
CPU: 0.48s Real: 0.51s RAM: 48188KB
$ time mono ./dictionary.exe
6009354 6009348 611297
36186112 159701682 23370001
CPU: 40.98s Real: 42.07s RAM: 1690224KB
Edit, after bugfixing the code for PHP, Python and JavaScript source, only JavaScriptCore, Rhino and Python3 that successfully complete the benchmark.
$ pacman -Qo `which jsc-3`
/usr/bin/jsc-3 is owned by webkitgtk 2.4.8-1
$ time jsc-3 object.js
6009354 6009348 611297
36186112 159701682 23370001
CPU: 148.09s Real: 121.65s RAM: 4666280KB
$ rhino < /dev/null
Rhino 1.7 release 4 2014 07 01
$ time rhino object.js
6009354 6009348 611297
36186112 159701682 23370001
CPU: 524.12s Real: 189.04s RAM: 4220872KB
$ python3 --version
Python 3.4.2
$ time python3 dictionary.py
6009354 6009348 611297
36186112 159701682 23370001
CPU: 157.71s Real: 163.37s RAM: 4335568KB
$ lua -v
Lua 5.2.3 Copyright (C) 1994-2013 Lua.org, PUC-Rio
$ time lua table.lua
6009354 6009348 611297
36186112 159701682 23370001
CPU: 101.73s Real: 106.45s RAM: 3079336KB
$ dart --version
Dart VM version: 1.8.5 (Tue Jan 13 12:44:14 2015) on "linux_x64"
$ time dart --old_gen_heap_size=4096 map.dart
6009354 6009348 611297
36186112 159701682 23370001
CPU: 133.35s Real: 133.64s RAM: 2538220KB
And the summary
/usr/bin/jsc-3 is owned by webkitgtk 2.4.8-1
$ time jsc-3 object.js
6009354 6009348 611297
36186112 159701682 23370001
CPU: 148.09s Real: 121.65s RAM: 4666280KB
$ rhino < /dev/null
Rhino 1.7 release 4 2014 07 01
$ time rhino object.js
6009354 6009348 611297
36186112 159701682 23370001
CPU: 524.12s Real: 189.04s RAM: 4220872KB
Python 3.4.2
$ time python3 dictionary.py
6009354 6009348 611297
36186112 159701682 23370001
CPU: 157.71s Real: 163.37s RAM: 4335568KB
$ lua -v
Lua 5.2.3 Copyright (C) 1994-2013 Lua.org, PUC-Rio
$ time lua table.lua
6009354 6009348 611297
36186112 159701682 23370001
CPU: 101.73s Real: 106.45s RAM: 3079336KB
$ dart --version
Dart VM version: 1.8.5 (Tue Jan 13 12:44:14 2015) on "linux_x64"
$ time dart --old_gen_heap_size=4096 map.dart
6009354 6009348 611297
36186112 159701682 23370001
CPU: 133.35s Real: 133.64s RAM: 2538220KB
Compiler / Interpreter | Language | Type Name | Data Structure | Compile Duration | Compile RAM | Runtime Duration | Runtime RAM | Correct? |
g++ (debug) | C++ | map | RB-Tree | 330 | 57764 | 141030 | 2558828 | |
g++ (-O2) | C++ | map | RB-Tree | 400 | 59672 | 65370 | 2558720 | |
g++ (debug) | C++ | unordered_map | Hash Table | 370 | 61804 | 45460 | 2478132 | |
g++ (-O2) | C++ | unordered_map | Hash Table | 410 | 62688 | 29160 | 2478148 | |
g++ (debug) | C++ | judySArray | 256-ary Radix Tree | 290 | 50088 | 27070 | 601604 | -2 |
g++ (-O2) | C++ | judySArray | 256-ary Radix Tree | 570 | 55192 | 16310 | 601600 | -2 |
g++ (debug) | C++ | cedar::da | Double Array Trie | 360 | 55344 | 37970 | 749144 | |
g++ (-O2) | C++ | cedar::da | Double Array Trie | 930 | 70436 | 20610 | 746352 | |
g++ (debug) | C++ | hattrie_t | HAT-Trie | 310 | 49844 | 41990 | 567048 | |
g++ (-O2) | C++ | hattrie_t | HAT-Trie | 540 | 55376 | 27030 | 567040 | |
clang++ (debug) | C++ | map | RB-Tree | 300 | 49828 | 138630 | 2558732 | |
clang++ (-O2) | C++ | map | RB-Tree | 390 | 53680 | 67430 | 2558720 | |
clang++ (debug) | C++ | unordered_map | Hash Table | 320 | 52148 | 45200 | 2478136 | |
clang++ (-O2) | C++ | unordered_map | Hash Table | 410 | 54828 | 30130 | 2478120 | |
clang++ (debug) | C++ | judySArray | 256-ary Radix Tree | 260 | 46872 | 27880 | 601636 | -2 |
clang++ (-O2) | C++ | judySArray | 256-ary Radix Tree | 460 | 51460 | 17270 | 601600 | -2 |
clang++ (debug) | C++ | cedar::da | Double Array Trie | 300 | 50012 | 38260 | 748868 | |
clang++ (-O2) | C++ | cedar::da | Double Array Trie | 480 | 53160 | 21950 | 749248 | |
clang++ (debug) | C++ | hattrie_t | HAT-Trie | 260 | 47908 | 44640 | 567016 | |
clang++ (-O2) | C++ | hattrie_t | HAT-Trie | 500 | 52532 | 27240 | 567040 | |
javac, java | Java | TreeMap | RB-Tree | 1280 | 62040 | 101050 | 3951752 | |
javac, java | Java | HashMap | Hash Table | 1210 | 63588 | 103540 | 4119388 | |
ruby | Ruby | Hash | Hash Table | 107910 | 3031872 | |||
jruby | Ruby | Hash | Hash Table | 180570 | 8740612 | |||
jsc-3 | JavaScript | Object | Unknown | 121650 | 4666280 | |||
rhino | JavaScript | Object | Unknown | 189040 | 4220872 | |||
go | Go | map | Hash Table | 150 | 34892 | 24100 | 2748784 | |
python3 | Python 3 | dict | Hash Table | 157710 | 4335568 | |||
pypy | Python 2 | dict | Hash Table | 79870 | 4034956 | |||
mcs | C# | Dictionary | Hash Table | 480 | 48188 | 40980 | 1690224 | |
lua | Lua | table | Hash Table | 101730 | 3079336 | |||
dart | Dart | Map | Hash Table | 133350 | 2538220 |
Note #2: again MozJS 2.4 got uncaught exception: out of memory when it uses 1.2GB of RAM
Note #3: LuaJIT 2.0.3 got not enough memory when it uses 1GB of RAM
Labels:
associative array
,
benchmark
,
c#
,
c++
,
cedar
,
clang
,
double array trie
,
hat-trie
,
hhvm
,
javascriptcore
,
judyarray
,
nodejs
,
php
,
pypy
,
python
,
ruby
Subscribe to:
Posts
(
Atom
)