Just want to share some table from initial chapter of my thesis (early 2012), it's about modification (added lots of compression) of HAT-Trie for DNS suffix blocking. These tables are from chapter IV, since it's the only exciting part about it .__.)/||. That time I didn't know about Cedar yet (of course it's 2013 XD). This is the list of benchmarked data structure:
The data structure name that marked with "*" sign are those who can be set as nested per subdomain (that is should be a map/associative array). The strings tested are C++'s std::string, Qt's QString, csubdom (compressed subdomain), strcsubdom (compressed subdomain, stored in std::string), clabels (compressed full domain), strclables (compressed full domain name). Compressed in this term are packed characters (from 8-bit to 5/6-bit so it would use less memory). This benchmark performed on Ubuntu 64-bit Linux 3.2, GCC 4.6.3, AMD Phenom X9850, 8GB RAM and non-SSD disk, compile flag: g++ -c -m64 -pipe -O2 -Wall -W.
Notes about that table header "insert" is a benchmark about inserting 2.1 million blacklisted domain names, after it's completed, the data structure erased and the insert operation repeated until 30 seconds passes. The "misses" benchmark about checking if 68.4k domains that doesn't exists on the blacklist, the operation repeated until 2 seconds passes. The "exists" benchmark is about rechecking blacklisted domain names in sorted order, repeated until 8 seconds passes. The "random" benchmark is about checking random blacklisted items, repeated until 20 seconds passes. The value there are number of milliseconds required per domain name. Last column on the table is average bytes required to store one domain name.
programming: the action or process of writing computer programs. | rants: speak or shout at length in a wild, [im]passioned way.
Showing posts with label hat-trie. Show all posts
Showing posts with label hat-trie. Show all posts
2015-02-27
2015-02-17
String Associative Array Benchmark
Today we will benchmark about associative array. Most associative array are implemented using hashtable or tree. We'll benchmark all built-in associative array for storing strings, that are C++'s map and unordered_map, Java's TreeMap and HashMap, PHP's Array, Ruby's Hash, JavaScript's Object, Go's map, Python's Dictionary, and C#'s Dictionary. The benchmark should only use built-in associative array, integer conversion, and string reverse function. The key to be tested are number-string, inverted, and reversed hex number from N to 0. The value to be stored are double precision floating point (if possible). The key must be searched first, when the key exists, the value incremented by the certain value. After building the associative array, iterate one by one and get the total of first digit of each value, total length of the key, and total number of entries. We also want to compare the performance some of the best data structures (in my experience, in terms of memory usage and speed ratio), that is 256-ary radix tree (judy-template), double-array trie (cedar), and HAT-trie (hat-trie). You can look up the source on my dropbox (folder: assoc), sorry, too lazy to use github ;3 This benchmark performed on 64-bit Linux, AMD A8-6600K, 16GB RAM, Non-SSD harddisk.
$ alias | grep 'alias time'
alias time='/usr/bin/time -f "\nCPU: %Us\tReal: %es\tRAM: %MKB"'
$ time --version
GNU time 1.7
$ g++ --version
g++ (GCC) 4.9.2 20141224 (prerelease)
$ time g++ -std=c++11 map.cpp
CPU: 0.33s Real: 0.39s RAM: 57764KB
$ time ./a.out
6009354 6009348 611297
36186112 159701682 23370001
CPU: 141.03s Real: 142.40s RAM: 2558828KB
$ time g++ -std=c++11 -O2 map.cpp
CPU: 0.40s Real: 0.45s RAM: 59672KB
$ time ./a.out
6009354 6009348 611297
36186112 159701682 23370001
$ alias | grep 'alias time'
alias time='/usr/bin/time -f "\nCPU: %Us\tReal: %es\tRAM: %MKB"'
$ time --version
GNU time 1.7
$ g++ --version
g++ (GCC) 4.9.2 20141224 (prerelease)
$ time g++ -std=c++11 map.cpp
CPU: 0.33s Real: 0.39s RAM: 57764KB
$ time ./a.out
6009354 6009348 611297
36186112 159701682 23370001
CPU: 141.03s Real: 142.40s RAM: 2558828KB
$ time g++ -std=c++11 -O2 map.cpp
CPU: 0.40s Real: 0.45s RAM: 59672KB
$ time ./a.out
6009354 6009348 611297
36186112 159701682 23370001
CPU: 65.37s     Real: 66.63s    RAM: 2558720KB
$ time g++ -std=c++11 unordered_map.cpp
CPU: 0.37s Real: 0.44s RAM: 61804KB
$ time ./a.out
6009354 6009348 611297
36186112 159701682 23370001
$ time g++ -std=c++11 unordered_map.cpp
CPU: 0.37s Real: 0.44s RAM: 61804KB
$ time ./a.out
6009354 6009348 611297
36186112 159701682 23370001
CPU: 45.46s     Real: 46.68s    RAM: 2478132KB
$ time g++ -std=c++11 -O2 unordered_map.cpp
CPU: 0.41s Real: 0.45s RAM: 62688KB
$ time ./a.out
6009354 6009348 611297
36186112 159701682 23370001
$ time g++ -std=c++11 -O2 unordered_map.cpp
CPU: 0.41s Real: 0.45s RAM: 62688KB
$ time ./a.out
6009354 6009348 611297
36186112 159701682 23370001
CPU: 29.16s     Real: 30.32s    RAM: 2478148KB
$ time g++ -std=c++11 judy.cpp
CPU: 0.29s Real: 0.34s RAM: 50088KB
$ time ./a.out
6009354 6009348 611297
36186112 159701681 23370000
CPU: 27.07s Real: 27.46s RAM: 601604KB
$ time g++ -std=c++11 -O2 judy.cpp
CPU: 0.57s Real: 0.62s RAM: 55192KB
$ time ./a.out
6009354 6009348 611297
36186112 159701681 23370000
$ time g++ -std=c++11 judy.cpp
CPU: 0.29s Real: 0.34s RAM: 50088KB
$ time ./a.out
6009354 6009348 611297
36186112 159701681 23370000
CPU: 27.07s Real: 27.46s RAM: 601604KB
$ time g++ -std=c++11 -O2 judy.cpp
CPU: 0.57s Real: 0.62s RAM: 55192KB
$ time ./a.out
6009354 6009348 611297
36186112 159701681 23370000
CPU: 16.31s     Real: 16.62s    RAM: 601600KB
$ time g++ -std=c++11 cedar.cpp
CPU: 0.36s Real: 0.40s RAM: 55344KB
$ time ./a.out
6009354 6009348 611297
36186112 159701682 23370001
$ time g++ -std=c++11 -O2 cedar.cpp
CPU: 0.93s Real: 0.97s RAM: 70436KB
$ time g++ -std=c++11 hat_trie.cpp
$ time g++ -std=c++11 -O2 hat_trie.cpp
$ clang --version
$ time clang++ -std=c++11 map.cpp
$ time clang++ -std=c++11 unordered_map.cpp
CPU: 0.32s Real: 0.36s RAM: 52148KB
$ time clang++ -std=c++11 -O2 unordered_map.cpp
CPU: 0.41s Real: 0.45s RAM: 54828KB
$ time clang++ -std=c++11 -w judy.cpp
CPU: 0.26s Real: 0.29s RAM: 46872KB
$ time clang++ -std=c++11 -O2 -w judy.cpp
CPU: 0.46s Real: 0.50s RAM: 51460KB
$ time clang++ -std=c++11 cedar.cpp
CPU: 0.30s Real: 0.35s RAM: 50012KB
$ time clang++ -std=c++11 -O2 cedar.cpp
CPU: 0.48s Real: 0.51s RAM: 53152KB
$ time clang++ -std=c++11 hat_trie.cpp
$ time clang++ -std=c++11 -O2 hat_trie.cpp
$ javac -version
$ time javac tree_map.java
$ time javac hash_map.java
 
 
$ jruby -version
Note #1: PHP 5.6.5, HHVM 3.5.0, Rubinius 2.5.2, NodeJS 0.10.35 failed to build the associative array within 300s.
Note #2: again MozJS 2.4 got uncaught exception: out of memory when it uses 1.2GB of RAM
Note #3: LuaJIT 2.0.3 got not enough memory when it uses 1GB of RAM
$ time g++ -std=c++11 cedar.cpp
CPU: 0.36s Real: 0.40s RAM: 55344KB
$ time ./a.out
6009354 6009348 611297
36186112 159701682 23370001
CPU: 37.97s     Real: 38.23s    RAM: 749144KB
$ time g++ -std=c++11 -O2 cedar.cpp
CPU: 0.93s Real: 0.97s RAM: 70436KB
$ time ./a.out 
6009354 6009348 611297
6009354 6009348 611297
36186112 159701682 23370001
CPU: 20.61s     Real: 20.89s    RAM: 746352KB
$ time g++ -std=c++11 hat_trie.cpp
CPU: 0.31s      Real: 0.36s     RAM: 49844KB
$ time ./a.out 
6009354 6009348 611297
6009354 6009348 611297
36186112 159701682 23370001
CPU: 41.99s     Real: 42.42s    RAM: 567048KB
$ time g++ -std=c++11 -O2 hat_trie.cpp
CPU: 0.54s      Real: 0.59s     RAM: 55376KB
$ time ./a.out 
 
 
6009354 6009348 611297
6009354 6009348 611297
36186112 159701682 23370001
CPU: 27.03s     Real: 27.37s    RAM: 567040KB 
$ clang --version
clang version 3.5.1 (tags/RELEASE_351/final)
$ time clang++ -std=c++11 map.cpp
CPU: 0.30s      Real: 0.33s     RAM: 49828KB
$ time ./a.out 
 
 
6009354 6009348 611297
6009354 6009348 611297
36186112 159701682 23370001 
CPU: 138.63s    Real: 140.16s   RAM: 2558732KB 
 
 
 
 
$ time clang++ -std=c++11 -O2 map.cpp
CPU: 0.39s      Real: 0.42s     RAM: 53680KB
$ time ./a.out 
 
 
6009354 6009348 611297
6009354 6009348 611297
36186112 159701682 23370001
CPU: 67.43s     Real: 68.73s    RAM: 2558720KB
$ time clang++ -std=c++11 unordered_map.cpp
CPU: 0.32s Real: 0.36s RAM: 52148KB
$ time ./a.out 
 
 
6009354 6009348 611297
6009354 6009348 611297
36186112 159701682 23370001 
 
CPU: 45.22s     Real: 46.46s    RAM: 2478136KB
$ time clang++ -std=c++11 -O2 unordered_map.cpp
CPU: 0.41s Real: 0.45s RAM: 54828KB
$ time ./a.out 
 
 
6009354 6009348 611297
6009354 6009348 611297
36186112 159701682 23370001 
CPU: 30.13s     Real: 31.30s    RAM: 2478120KB
$ time clang++ -std=c++11 -w judy.cpp
CPU: 0.26s Real: 0.29s RAM: 46872KB
$ time ./a.out 
 
 
6009354 6009348 611297
6009354 6009348 611297
36186112 159701681 23370000 
 
CPU: 27.88s     Real: 28.21s    RAM: 601636KB
$ time clang++ -std=c++11 -O2 -w judy.cpp
CPU: 0.46s Real: 0.50s RAM: 51460KB
$ time ./a.out 
 
 
6009354 6009348 611297
6009354 6009348 611297
36186112 159701681 23370000 
 
CPU: 17.27s     Real: 17.64s    RAM: 601600KB
$ time clang++ -std=c++11 cedar.cpp
CPU: 0.30s Real: 0.35s RAM: 50012KB
$ time ./a.out 
 
 
6009354 6009348 611297
6009354 6009348 611297
36186112 159701682 23370001 
 
CPU: 38.26s     Real: 38.54s    RAM: 748868KB
$ time clang++ -std=c++11 -O2 cedar.cpp
CPU: 0.48s Real: 0.51s RAM: 53152KB
$ time ./a.out 
 
 
6009354 6009348 611297
6009354 6009348 611297
36186112 159701682 23370001 
 
CPU: 21.95s     Real: 22.19s    RAM: 749248KB 
 
$ time clang++ -std=c++11 hat_trie.cpp
CPU: 0.26s      Real: 0.29s     RAM: 47700KB
$ time ./a.out 
 
 
6009354 6009348 611297
6009354 6009348 611297
36186112 159701682 23370001 
 
CPU: 44.64s     Real: 45.04s    RAM: 567016KB 
 
$ time clang++ -std=c++11 -O2 hat_trie.cpp
CPU: 0.50s      Real: 0.54s     RAM: 52532KB
$ time ./a.out 
 
 
 
 
6009354 6009348 611297
6009354 6009348 611297
36186112 159701682 23370001 
 
CPU: 27.24s     Real: 27.57s    RAM: 567040KB 
 
$ javac -version
javac 1.7.0_75
$ time javac tree_map.java
CPU: 1.28s      Real: 0.86s     RAM: 62040KB
$ time java tree_map
6009354 6009348 611297 
 
 
36186112 159701682 23370001
36186112 159701682 23370001
CPU: 233.50s    Real: 101.05s   RAM: 3951752KB
$ time javac hash_map.java
CPU: 1.21s      Real: 0.84s     RAM: 63588KB
$ time java hash_map
6009354 6009348 611297
36186112 159701682 23370001
CPU: 345.03s    Real: 103.54s   RAM: 4119388KB
$ ruby --version 
 
 
ruby 2.2.0p0 (2014-12-25 revision 49005) [x86_64-linux]
ruby 2.2.0p0 (2014-12-25 revision 49005) [x86_64-linux]
$ time ruby hash.rb
6009354 6009348 611297
36186112 159701682 23370001 
 
CPU: 107.91s    Real: 109.66s   RAM: 3031872KB
$ jruby -version
jruby 9.0.0.0.pre1 (2.2.0p0) 2015-01-20 d537cab OpenJDK 64-Bit Server VM 24.75-b04 on 1.7.0_75-b13 +jit [linux-amd64]
$ time jruby -J-Xmx12000M -J-Djruby.compile.mode=FORCE hash.rb
6009354 6009348 611297 
 
 
 
 
36186112 159701682 23370001
CPU: 421.81s    Real: 180.57s   RAM: 8740612KB
CPU: 0.15s      Real: 0.20s     RAM: 34892KB
$ time ./map
6009354 6009348 611297
36186112 159701682 23370001
CPU: 24.10s     Real: 25.01s    RAM: 2748784KB
$ pypy --version
Python 2.7.8 (c6ad44ecf5d8, Nov 18 2014, 18:04:31) [PyPy 2.4.0 with GCC 4.9.2]
$ time pypy dictionary.py
(6009354, 6009348, 611297)
(36186112, 159701682, 23370001)
CPU: 79.87s Real: 84.57s RAM: 4034956KB
(36186112, 159701682, 23370001)
CPU: 79.87s Real: 84.57s RAM: 4034956KB
$ mcs --version
Mono C# compiler version 3.12.0.0
$ time mcs dictionary.cs
CPU: 0.48s      Real: 0.51s     RAM: 48188KB
$ time mono ./dictionary.exe
6009354 6009348 611297
36186112 159701682 23370001
CPU: 40.98s     Real: 42.07s    RAM: 1690224KB
Edit, after bugfixing the code for PHP, Python and JavaScript source, only JavaScriptCore, Rhino and Python3 that successfully complete the benchmark.
$ pacman -Qo `which jsc-3`
/usr/bin/jsc-3 is owned by webkitgtk 2.4.8-1
$ time jsc-3 object.js
6009354 6009348 611297
36186112 159701682 23370001
CPU: 148.09s Real: 121.65s RAM: 4666280KB
$ rhino < /dev/null
Rhino 1.7 release 4 2014 07 01
$ time rhino object.js
6009354 6009348 611297
36186112 159701682 23370001
CPU: 524.12s Real: 189.04s RAM: 4220872KB
$ python3 --version
Python 3.4.2
$ time python3 dictionary.py
6009354 6009348 611297
36186112 159701682 23370001
CPU: 157.71s Real: 163.37s RAM: 4335568KB
$ lua -v
Lua 5.2.3 Copyright (C) 1994-2013 Lua.org, PUC-Rio
$ time lua table.lua
6009354 6009348 611297
36186112 159701682 23370001
CPU: 101.73s Real: 106.45s RAM: 3079336KB
$ dart --version
Dart VM version: 1.8.5 (Tue Jan 13 12:44:14 2015) on "linux_x64"
$ time dart --old_gen_heap_size=4096 map.dart
6009354 6009348 611297
36186112 159701682 23370001
CPU: 133.35s Real: 133.64s RAM: 2538220KB
And the summary
/usr/bin/jsc-3 is owned by webkitgtk 2.4.8-1
$ time jsc-3 object.js
6009354 6009348 611297
36186112 159701682 23370001
CPU: 148.09s Real: 121.65s RAM: 4666280KB
$ rhino < /dev/null
Rhino 1.7 release 4 2014 07 01
$ time rhino object.js
6009354 6009348 611297
36186112 159701682 23370001
CPU: 524.12s Real: 189.04s RAM: 4220872KB
Python 3.4.2
$ time python3 dictionary.py
6009354 6009348 611297
36186112 159701682 23370001
CPU: 157.71s Real: 163.37s RAM: 4335568KB
$ lua -v
Lua 5.2.3 Copyright (C) 1994-2013 Lua.org, PUC-Rio
$ time lua table.lua
6009354 6009348 611297
36186112 159701682 23370001
CPU: 101.73s Real: 106.45s RAM: 3079336KB
$ dart --version
Dart VM version: 1.8.5 (Tue Jan 13 12:44:14 2015) on "linux_x64"
$ time dart --old_gen_heap_size=4096 map.dart
6009354 6009348 611297
36186112 159701682 23370001
CPU: 133.35s Real: 133.64s RAM: 2538220KB
| Compiler / Interpreter | Language | Type Name | Data Structure | Compile Duration | Compile RAM | Runtime Duration | Runtime RAM | Correct? | 
| g++ (debug) | C++ | map | RB-Tree | 330 | 57764 | 141030 | 2558828 | |
| g++ (-O2) | C++ | map | RB-Tree | 400 | 59672 | 65370 | 2558720 | |
| g++ (debug) | C++ | unordered_map | Hash Table | 370 | 61804 | 45460 | 2478132 | |
| g++ (-O2) | C++ | unordered_map | Hash Table | 410 | 62688 | 29160 | 2478148 | |
| g++ (debug) | C++ | judySArray | 256-ary Radix Tree | 290 | 50088 | 27070 | 601604 | -2 | 
| g++ (-O2) | C++ | judySArray | 256-ary Radix Tree | 570 | 55192 | 16310 | 601600 | -2 | 
| g++ (debug) | C++ | cedar::da | Double Array Trie | 360 | 55344 | 37970 | 749144 | |
| g++ (-O2) | C++ | cedar::da | Double Array Trie | 930 | 70436 | 20610 | 746352 | |
| g++ (debug) | C++ | hattrie_t | HAT-Trie | 310 | 49844 | 41990 | 567048 | |
| g++ (-O2) | C++ | hattrie_t | HAT-Trie | 540 | 55376 | 27030 | 567040 | |
| clang++ (debug) | C++ | map | RB-Tree | 300 | 49828 | 138630 | 2558732 | |
| clang++ (-O2) | C++ | map | RB-Tree | 390 | 53680 | 67430 | 2558720 | |
| clang++ (debug) | C++ | unordered_map | Hash Table | 320 | 52148 | 45200 | 2478136 | |
| clang++ (-O2) | C++ | unordered_map | Hash Table | 410 | 54828 | 30130 | 2478120 | |
| clang++ (debug) | C++ | judySArray | 256-ary Radix Tree | 260 | 46872 | 27880 | 601636 | -2 | 
| clang++ (-O2) | C++ | judySArray | 256-ary Radix Tree | 460 | 51460 | 17270 | 601600 | -2 | 
| clang++ (debug) | C++ | cedar::da | Double Array Trie | 300 | 50012 | 38260 | 748868 | |
| clang++ (-O2) | C++ | cedar::da | Double Array Trie | 480 | 53160 | 21950 | 749248 | |
| clang++ (debug) | C++ | hattrie_t | HAT-Trie | 260 | 47908 | 44640 | 567016 | |
| clang++ (-O2) | C++ | hattrie_t | HAT-Trie | 500 | 52532 | 27240 | 567040 | |
| javac, java | Java | TreeMap | RB-Tree | 1280 | 62040 | 101050 | 3951752 | |
| javac, java | Java | HashMap | Hash Table | 1210 | 63588 | 103540 | 4119388 | |
| ruby | Ruby | Hash | Hash Table | 107910 | 3031872 | |||
| jruby | Ruby | Hash | Hash Table | 180570 | 8740612 | |||
| jsc-3 | JavaScript | Object | Unknown | 121650 | 4666280 | |||
| rhino | JavaScript | Object | Unknown | 189040 | 4220872 | |||
| go | Go | map | Hash Table | 150 | 34892 | 24100 | 2748784 | |
| python3 | Python 3 | dict | Hash Table | 157710 | 4335568 | |||
| pypy | Python 2 | dict | Hash Table | 79870 | 4034956 | |||
| mcs | C# | Dictionary | Hash Table | 480 | 48188 | 40980 | 1690224 | |
| lua | Lua | table | Hash Table | 101730 | 3079336 | |||
| dart | Dart | Map | Hash Table | 133350 | 2538220 | 
Note #2: again MozJS 2.4 got uncaught exception: out of memory when it uses 1.2GB of RAM
Note #3: LuaJIT 2.0.3 got not enough memory when it uses 1GB of RAM
Labels:
associative array
                                              ,
                                            
benchmark
                                              ,
                                            
c#
                                              ,
                                            
c++
                                              ,
                                            
cedar
                                              ,
                                            
clang
                                              ,
                                            
double array trie
                                              ,
                                            
hat-trie
                                              ,
                                            
hhvm
                                              ,
                                            
javascriptcore
                                              ,
                                            
judyarray
                                              ,
                                            
nodejs
                                              ,
                                            
php
                                              ,
                                            
pypy
                                              ,
                                            
python
                                              ,
                                            
ruby
Subscribe to:
Comments
                                      (
                                      Atom
                                      )
                                    





