2017-05-22

Go-Redis vs RediGo (also Aerospike)

This is an old benchmark result that test Redis and Aerospike, both are in-memory database, I did this about December last year, that I used to test Redis agains Aerospike for cases of storing random session per request:

Redis (redigo)
Transactions:                  52343 hits 
Availability:                 100.00 % 
Elapsed time:                   9.16 secs 
Data transferred:              17.02 MB 
Response time:                  0.04 secs 
Transaction rate:            5714.30 trans/sec  (1654 worst >1M uQ sess)
Throughput:                     1.86 MB/sec 
Concurrency:                  252.55 
Successful transactions:       52343 
Failed transactions:               0 
Longest transaction:            1.13 
Shortest transaction:           0.00

Aerospike (aerospike-client-go)
Transactions:                  80806 hits
Availability:                 100.00 %
Elapsed time:                   9.71 secs
Data transferred:              26.28 MB
Response time:                  0.03 secs
Transaction rate:            8321.94 trans/sec (8999 best, 7769 worst)
Throughput:                     2.71 MB/sec
Concurrency:                  251.91
Successful transactions:       80806
Failed transactions:               0
Longest transaction:            1.17
Shortest transaction:           0.00

Redis (go-redis)
Transactions:                  91187 hits 
Availability:                 100.00 % 
Elapsed time:                   9.95 secs 
Data transferred:              29.65 MB 
Response time:                  0.03 secs 
Transaction rate:            9164.52 trans/sec (3536 worst >1M uQ sess)
Throughput:                     2.98 MB/sec 
Concurrency:                  252.70 
Successful transactions:       91187 
Failed transactions:               0 
Longest transaction:            0.20 
Shortest transaction:           0.00 

The bad part about Redis (that uses SkipList), more data we store, it slows down faster, in this case 1 million sessions stored slows Redis down by more than 60%, while Aerospike only slowed down by 10%).

Redis 3.2.1 vs ScyllaDB 1.7RC2

Since Scylla still have no secondary index, all I can do to use Scylla is to replace Redis for storing user login sessions, this benchmark only test read queries (queries that always returns zero request because the record does not exists) and the result are:

# Redis 
$ hey -c 255 -n 255000 http://localhost:3001
3089 requests done.
7217 requests done.
11691 requests done.
*snip*
241822 requests done.
246305 requests done.
250697 requests done.
All requests done.

Summary:
  Total:        29.5162 secs
  Slowest:      0.1647 secs
  Fastest:      0.0003 secs
  Average:      0.0294 secs
  Requests/sec: 8639.3205
  Total data:   2732835000 bytes
  Size/request: 10717 bytes

Status code distribution:
  [200] 255000 responses

Response time histogram:
  0.000 [1]     |
  0.017 [4084]  |∎
  0.033 [194502]|∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  0.050 [54311] |∎∎∎∎∎∎∎∎∎∎∎
  0.066 [1812]  |
  0.082 [65]    |
  0.099 [7]     |
  0.115 [122]   |
  0.132 [48]    |
  0.148 [25]    |
  0.165 [23]    |

Latency distribution:
  10% in 0.0231 secs
  25% in 0.0255 secs
  50% in 0.0286 secs
  75% in 0.0325 secs
  90% in 0.0370 secs
  95% in 0.0404 secs
  99% in 0.0487 secs

# ScyllaDB best response time
$ hey -c 255 -n 255000 http://localhost:3001
2114 requests done.
4874 requests done.
7714 requests done.
*snip*
247202 requests done.
249898 requests done.
252610 requests done.
All requests done.

Summary:
  Total:        48.5436 secs
  Slowest:      0.2649 secs
  Fastest:      0.0013 secs
  Average:      0.0483 secs
  Requests/sec: 5253.0127
  Total data:   2732835000 bytes
  Size/request: 10717 bytes

Status code distribution:
  [200] 255000 responses

Response time histogram:
  0.001 [1]     |
  0.028 [6804]  |∎∎
  0.054 [176673]|∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  0.080 [66748] |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  0.107 [3728]  |∎
  0.133 [470]   |
  0.159 [250]   |
  0.186 [46]    |
  0.212 [79]    |
  0.239 [144]   |
  0.265 [57]    |

Latency distribution:
  10% in 0.0334 secs
  25% in 0.0399 secs
  50% in 0.0466 secs
  75% in 0.0552 secs
  90% in 0.0636 secs
  95% in 0.0699 secs

  99% in 0.0899 secs

# ScyllaDB best Req/s
$ hey -c 255 -n 255000 http://localhost:3001
2188 requests done.
4910 requests done.
7019 requests done.
*snip*
244547 requests done.
249813 requests done.
254894 requests done.
All requests done.

Summary:
  Total:        42.0725 secs
  Slowest:      8.0907 secs
  Fastest:      0.0002 secs
  Average:      0.0418 secs
  Requests/sec: 6060.9647
  Total data:   2732835000 bytes
  Size/request: 10717 bytes

Status code distribution:
  [200] 255000 responses

Response time histogram:
  0.000 [1]     |
  0.809 [254744]|∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  1.618 [0]     |
  2.427 [0]     |
  3.236 [0]     |
  4.045 [0]     |
  4.854 [0]     |
  5.664 [0]     |
  6.473 [0]     |
  7.282 [0]     |
  8.091 [255]   |

Latency distribution:
  10% in 0.0069 secs
  25% in 0.0183 secs
  50% in 0.0347 secs
  75% in 0.0470 secs
  90% in 0.0573 secs
  95% in 0.0640 secs
  99% in 0.0843 secs

This is the example code of the main function using gotro framework that used to do this benchmark:


//// Testing Redis:
//login_conn := Rd.NewRedisSession(``, ``, 1, `session::`)
//global_conn := Rd.NewRedisSession(``, ``, 3, `global::`)

// Testing Scylla:
login_conn:= Sc.NewScyllaSession(`127.0.0.1`, `session`, `login`, ``, ``)
global_conn := Sc.NewScyllaSession(`127.0.0.1`, `session`, `global`, ``, ``)

// see example
W.InitSession(`SK`, 12*time.Hour, 6*time.Hour, *login_conn, *global_conn)
W.Mailers = ...
W.Assets = ...
W.Webmasters = ...
W.Routes = ...
server := W.NewEngine(DEBUG_MODE, false, `test`, ROOT_DIR)
server.StartServer(LISTEN_ADDR)

And just like before, after doing this intensive benchmark, the Ubuntu showed an error for Scylla:


The bad part about Redis is the scalability stuck on single core, if you add more server the write will not scale, so Scylla is better replacement if you want to do horizontal scaling.

2017-05-19

PostgreSQL 9.6.2 vs ScyllaDB 1.7RC2

Since CockroachDB 1.0 not yet performant (ain't got time to wait), today we're gonna test PostgreSQL 9.6.2 vs ScyllaDB 1.7RC2 on Ubuntu XFS filesystem.

test1: postgresql
INSERT: 34.667685316s (3.47 ms/op)
UPDATE: 35.117617526s (3.51 ms/op)
SELECT: 47.529755777s (0.73 ms/op)
CPU: 35.14s     Real: 117.64s   RAM: 58 544 KB

test2: postgresql jsonb
INSERT: 33.861673279s (3.39 ms/op)
UPDATE: 34.038996914s (3.40 ms/op)
SELECT: 45.340834079s (0.70 ms/op)
CPU: 33.62s     Real: 113.58s   RAM: 58 140 KB

test4: scylladb
INSERT: 2.133985799s (0.21 ms/op)
UPDATE: 2.167973712s (0.22 ms/op)
SELECT: 2m24.804415353s (2.22 ms/op)
CPU: 41.29s     Real: 152.57s   RAM: 79 708 KB

Hmm.. This is weird, because the difference between this and previous benchmark are:

LabelThisPrevious
Operating System Ubuntu 17.04 Manjaro
Kernel 4.10.0-19-generic
tuned by scylla_setup
aufs_friendly 4.10.13-1
PostgreSQL config:
effective_cache
shared_buffers
work_mem
ubuntu default:
4G/128M/4M
modified:
2G/2G/16M
PostgreSQL total time 113.58s 116.34s
ScyllaDB  official deb 1.7RC2 official docker 1.6.4
ScyllaDB config 8GB / 8 / XFS 4GB / 4 / EXT4
ScyllaDB total time 152.57s 141.34s

Probably the kernel factor? Oh yeah, you can get the source on github (you can create PR if there's bug). Now let's test it in parallel with one third of data (only show first 3 and last 3 result):

test1: postgresql
I-18: (6.17 ms/op: 101)
I-12: (6.16 ms/op: 101)
I-14: (6.19 ms/op: 101)
I-32: (157.80 ms/op: 101)
I-05: (167.31 ms/op: 101)
I-31: (168.23 ms/op: 101)
INSERT: 16.99355453s (5.10 ms/op)
U-13: (5.92 ms/op: 101)
U-16: (5.97 ms/op: 101)
U-04: (6.00 ms/op: 101)
U-21: (1312.46 ms/op: 101)
U-28: (1333.33 ms/op: 101)
U-20: (1333.60 ms/op: 101)
UPDATE: 2m14.695128106s (40.41 ms/op)
S-24: (14.56 ms/op: 139)
S-69: (17.86 ms/op: 115)
S-78: (24.18 ms/op: 88)
S-06: (18.14 ms/op: 556)
S-46: (18.54 ms/op: 556)
S-37: (1427.27 ms/op: 91)
SELECT: 2m9.888985893s (5.98 ms/op: 21716)
CPU: 13.07s     Real: 281.90s   RAM: 59 072 KB

test2: postgresql jsonb
I-03: (6.23 ms/op: 101)
I-10: (6.28 ms/op: 101)
I-04: (6.30 ms/op: 101)
I-08: (157.54 ms/op: 101)
I-29: (225.82 ms/op: 101)
I-22: (314.39 ms/op: 101)
INSERT: 31.754559744s (9.53 ms/op)
U-23: (6.45 ms/op: 101)
U-13: (6.51 ms/op: 101)
U-04: (6.54 ms/op: 101)
U-05: (1287.63 ms/op: 101)
U-01: (1287.77 ms/op: 101)
U-26: (1953.67 ms/op: 101)
UPDATE: 3m17.321405467s (59.20 ms/op)
S-27: (27.54 ms/op: 124)
S-21: (21.89 ms/op: 159)
S-29: (30.55 ms/op: 115)
S-48: (323.69 ms/op: 417)
S-05: (202.41 ms/op: 667)
S-03: (121.61 ms/op: 1111)
SELECT: 2m15.109162326s (6.22 ms/op: 21716)
CPU: 13.78s     Real: 364.50s   RAM: 56 688 KB

test4: scylladb
I-17: (1.59 ms/op: 101)
I-12: (1.59 ms/op: 101)
I-30: (1.60 ms/op: 101)
I-13: (1.65 ms/op: 101)
I-22: (1.65 ms/op: 101)
I-06: (1.65 ms/op: 101)
INSERT: 166.992399ms (0.05 ms/op)
U-06: (1.60 ms/op: 101)
U-12: (1.61 ms/op: 101)
U-29: (1.62 ms/op: 101)
U-31: (1.68 ms/op: 101)
U-30: (1.68 ms/op: 101)
U-11: (1.68 ms/op: 101)
UPDATE: 170.240627ms (0.05 ms/op)
S-79: (52.05 ms/op: 86)
S-74: (50.06 ms/op: 98)
S-75: (53.78 ms/op: 96)
S-03: (12.18 ms/op: 1111)
S-02: (8.43 ms/op: 1666)
S-42: (8.53 ms/op: 1666)
SELECT: 14.678651323s (0.68 ms/op: 21716)
CPU: 16.92s     Real: 18.08s    RAM: 76 824 KB

WTF! These numbers are blazing fast!
Too bad that the gocql or probably the scylla-server sometimes refused to connect:

2017/05/19 14:16:28 gocql: unable to create session: unable to discover protocol version: dial tcp 127.0.0.1:9042: i/o timeout
2017/05/19 14:16:33 gocql: unable to create session: unable to discover protocol version: dial tcp 127.0.0.1:9042: i/o timeout
2017/05/19 14:20:30 gocql: unable to create session: unable to discover protocol version: dial tcp 127.0.0.1:9042: i/o timeout

2017/05/19 14:18:28 gocql: unable to create session: unable to discover protocol version: dial tcp 127.0.0.1:9042: getsockopt: connection refused
2017/05/19 14:18:44 gocql: unable to dial control conn 127.0.0.1: dial tcp 127.0.0.1:9042: i/o timeout
2017/05/19 14:18:44 gocql: unable to create session: control: unable to connect to initial hosts: dial tcp 127.0.0.1:9042: i/o timeout

2017/05/19 14:20:34 gocql: unable to create session: unable to discover protocol version: dial tcp 127.0.0.1:9042: i/o timeout
2017/05/19 14:20:37 gocql: unable to create session: unable to discover protocol version: dial tcp 127.0.0.1:9042: i/o timeout 

2017/05/19 14:27:46 gocql: unable to create session: unable to discover protocol version: dial tcp 127.0.0.1:9042: i/o timeout

Or when using cqlsh 127.0.0.1:

Connection error: ('Unable to connect to any servers', {'127.0.0.1': error(None, "Tried connecting to [('127.0.0.1', 9042)]. Last error: timed out")})
Connection error: ('Unable to complete the operation against any hosts', {})
Connection error: ('Unable to connect to any servers', {'127.0.0.1': error(4, "Tried connecting to [('127.0.0.1', 9042)]. Last error: Interrupted system call")})

Other those issue, I think ScyllaDB is freakingly awesome!

EDIT: this issue only happened after doing these intensive benchmark, after restarting this no longer happened

2017-05-15

PostgreSQL 9.6.2 vs CockroachDB 1.0 vs ScyllaDB 1.6.4

New kids on the block, multi-master database that recently released 1.0, here's some microbenchmark result:

N = 999

test1: postgresql
INSERT: 3.442695947s (3.45 ms/op)
UPDATE: 3.912135754s (3.92 ms/op)
SELECT: 3.408927374s (0.52 ms/op)
CPU: 2.62s      Real: 11.07s    RAM: 57 864 KB

test2: postgresql jsonb
INSERT: 3.270218052s (3.27 ms/op)
UPDATE: 3.796453051s (3.80 ms/op)
SELECT: 3.209289448s (0.49 ms/op)
CPU: 2.33s      Real: 10.57s    RAM: 58 680 KB

test3: cockroachdb
INSERT: 7.495245970s (7.50 ms/op)
UPDATE: 8.249719113s (8.26 ms/op)
SELECT: 16m8.372273781s (148.34 ms/op)
CPU: 2.50s      Real: 986.17s   RAM: 58 340 KB

test4: scylladb
INSERT: 150.117719ms (0.15 ms/op)
UPDATE: 147.339553ms (0.15 ms/op)
SELECT: 5.422713068s (0.83 ms/op)
CPU: 4.06s      Real: 7.76s     RAM: 76 764 KB

N = 9999

test2: postgresql jsonb
INSERT: 36.012436525s (3.60 ms/op)
UPDATE: 35.902222429s (3.59 ms/op)
SELECT: 44.119970723s (0.68 ms/op)
CPU: 32.30s     Real: 116.34s   RAM: 58 632 KB

test4: scylladb
INSERT: 1.518285796s (0.15 ms/op)
UPDATE: 1.542542984s (0.15 ms/op)
SELECT: 2m16.29325852s (2.09 ms/op)
CPU: 41.55s     Real: 141.34s   RAM: 76 712 KB

This is shocking for me that CockroachDB 2x-19x slower than PostgreSQL, so I file a bug report and one for scylla (slow query on larger datasets).
This benchmark performed on 64-bit ArchLinux, i7-4720HQ, 16GB RAM, 256GB SSD Samsung. You can get the source here. Note that ScyllaDB requires XFS, but I use EXT4 filesystem.

PostgreSQL
+ battletested for 20 years
+ schema-free (via JSONB, it has indexes :3)
+ triggers, joins, language extensions (eg. pl/v8, pl/go, pl/ruby, etc)
- no multimaster replication, except if you use PostgresXL or Postgres-X2

CockroachDB
+ survive Aphyr's Jepsen
+ autoscaling, autohealing, autobalancing
+ only 1 binary file (Go power ^^)
- seriously slow on every part of this benchmark
- no BLOB! (as per 2017-05-15)

ScyllaDB
+ Cassandra rewritten in C++
+ autoscaling, autohealing, autobalancing
+ blazing fast for insert and update benchmark (not sure if it's persisted to disk though)
- no secondary index and serial/auto-increment (as per 2017-05-15)
- only support Ubuntu, Debian, RHEL (a bit challenging to compile on another OS because it's depends on old thrift and boost library)
- communication by default without authentication, this is bad if you don't have any private network (eg. host it on a public cloud), you must enable internode-ecnryption and put a firewall to allow only certain host access exposed ports.

2017-05-11

TechEmpower Framework Benchmark Round 14

New benchmark result is out, as usual the important part is the data-update benchmark:


At that chart, the top ranking language are: Kotlin, C, Java, C++, Go, Perl, Javascript, Scala, C#; and for the database: MySQL, PostgreSQL, MongoDB.

Also the other benchmark that reflect real world case is multiple-queries:
On that benchmmark, the top performer programming language are: Dart, C++, Java, C, Go, Kotlin, Javascript, Scala, Ruby, and Ur; and the database: MongoDB, PostgreSQL, MySQL. You can see the previous result here, and here.


2017-02-01

Elixir vs Golang

Rather than debate between newbies and expert in only one language, let's find out the pros and cons between Elixir and Go:
  1. The Syntax and Learning Curve
    In Go you can start after studying about 1 day since the syntax really similar to C (most universities taught C-family language), you can feel productive right away.
    In Elixir you'll need more than just 1 day (and obviously exponentially more to get the feel in Erlang unless you've learned about Prolog and LISP before), the syntax is somehow similar to Ruby, but you also required to learn about FP concepts (just like another functional language: Haskell, LISP, Clojure, F#) that could make you a better programmer.
  2. Concurrency and Deployment
    In Go you can achieve faster concurrency for single machine, but at cost of memory usage for the same amount of unprocessed light-thread (about 2-2.6x, see the edit history of previous link). If you need to need more than one machine, you must do it manually (but it's easy since Go statically linked: just a simple scp and executing service script would do).
    In Elixir you can have distributed concurrency, as described by many Erlang expert, BEAM is a 30 years old battle-tested virtual machine, that has these built-in advantages:
    1. Lightweight user-space threads (Goroutines requires more memory)
    2. Built-in distribution and failure detection (not sure what's the comparable library in Go)
    3. Reliability-oriented standard library (in Go you must check every error)
    4. Hot code swapping (use endless in Go to achieve zero downtime)
    Definitely you'll need time to master.
  3. Raw Performance
    In Go you got raw-performance, similar to Java, but more memory efficient, for any CPU-bound tasks, you should prefer Go instead of anything that currently has slower implementation (Javascript, PHP, Python, Perl, Ruby, Erlang/Elixir), see the 16k concurrent user column.
    In Elixir or any other BEAM language, since the light-thread have smaller memory usage, you can handle more process at the same time.
  4. Hiring
    Since Go are relatively more popular (because it's easier to learn) in terms of number of job postings I've encountered, TIOBE index (Jan 2017: Go #13, Erlang #44, Elixir #66), GitHub popularity go vs elixir, or Spectrum (Go #10. Erlang #35); than other BEAM-based language (especially Erlang), if you are PM/VPE with tight deadline, I believe Go is the better choice at this moment
So what you should use for your next project? It's always depends on what's the use case (right tool/person for the right job) and the deadline, there are no silver bullet. And no I don't intent to start a flame war.

2017-01-02

If Programming Language were Humans

Taken from here and nixCraft page (click there to see larger text).


  • Python: my formatting is my syntax!
  • Javascript: I'm technically functional
  • PHP: can I GET guys anything?
  • Haskell: (lambda x.x am pure)(i)
  • C/C++: If you're not allocating memory, you're not living!
  • C#: I am nothing like my father.
  • Erlang: Destroy one of my processes and I will only grow stronger
  • Elixir: I Draw my Eldritch Power From all my process-chans
  • Rust: I'll be designated driver tonight!
    Programmer: we know
    Rust: Don't talk while eatingyou could choke and die haha
  • Fortran: I am unparalled in numerical ^ scientific computation!
  • Matlab: Come with me if you want to research
  • Lua: I'll take things from here
  • Lisp Dialects: Trust in the recursion
    Racket: learning is fun

Ah there are some updates!























2017-01-01

Redis GUI

Redis is one of full featured in-memory database with optional persistence and replication. Redis support 5 kind of data types: key-value (SET, GET), hashtable (HSET, HGET), linkedlist (L/RPUSH, L/RPOP), sets (SADD, SREM) and sorted/scored-sets (ZADD, ZREM).



Looking for GUI for redis? Thy these apps, from the best to the least:

Redis React

Built using Mono + ReactJS. Just download and Run (install Mono first if you are using Linux).

Redis Desktop Manager

Built using C++ and Qt5.

yaourt --needed --noconfirm -S --force redis-desktop-manager

FastoRedis

A fork of FastoNoSql, I don't know what this one built with, since the github repository doesn't have the source.

Redis Commander

Built using NodeJS

sudo npm install -g redis-commander

Rebrow

Built using Python2 and Flask Framework. Just clone the repository, install its dependencies and run.

Btw, happy new year :3