Getting started with InfluxDB

Today we're gonna learn about InfluxDB, a time series database that already been a standard for open source metric storage, a part of TICK stack (telegraf -- a metric collector like datadog-agent/fluentd, influx -- the database, chronograf -- visualizer like grafana, kapacitor -- an alert manager like prometheus alert-manager also for ETL). To install influx use this steps from this link (there's also docker). What's the cons of influx compared to other solution? Datadog is (very very) expensive, Prometheus seems too kube-oriented, Influx open source doesn't support HA and clustering, that's why I would rather use Clickhouse for time series / log collection (as data sink) and aggregate those logs as metrics (materialized view) and copy periodically to faster database like Tarantool. Influx have 2 query syntax, one is SQL-like called InfluxQL and other one Javascript+FP-like called Flux (it has |> pipe like most FP). Here's the example of docker and the query language:

docker run -d -e INFLUXDB_ADMIN_USER:user1 -e INFLUXDB_ADMIN_PASSWORD:pass1 --name influxdb1 influxdb

docker exec -it influxdb1 influx -host 127.0.0.1 -port 8086 -username user1 -password pass1

show databases
create database db1
use db1

-- show all tables (=measurements)
show measurements

-- tag/index always string, field can be float, int, bool, etc
insert into table1,tag1=a,tag2=b field1=1,field2=2

-- if not set, default time is in nanosecond of insert time
select * from "table1"
select * from table1 where tag1='a'

-- describe columns from all table:
show tag keys
show field keys from "table1"

-- select distinct tag from table1
show tag values from "table1" with key in ("tag1")

-- automatically delete after N days, default 0 = never delete
-- shard by N hours, default 168 hours (a week)
show retention policies

-- like partition in other database
show shards
show shard groups

I guess that's it what you need to know to get started with InfluxDB. If you're looking one that comparable with Clickhouse, you can check TDEngine or TiFlash.

Database Patterns of Microservices

When you need microservice? When you have multiple business domain (not domain of DNS), that are best to be splitted, managed, and deployed separately. If your business domain so small and the team is small, it's better to use modular monolith instead, since using Microservice adds a lot of operational complexity (especially if you are using Kubernetes).

These are database patterns I got from Kindson presentation.

1. Private database per service
   this is the most common pattern, every domain/service must have their own database, this has some benefit:
   + the developers won't be tempted to join across domains that could make the codebase hard to refactor if someday need to be splitted to microservice/modular approach
   + easier for new developer that joining the team, since you don't need to know whole ER diagram, just a small segment that related to the service he/she managed
   - more complicated for analytics use case because you can't do JOIN, but this can be solved using distributed sql query engine like Trino
   + each database can scale and migrate independently (no downtimes especially when you are using database that require locking on migration like MySQL)
   - this causes another problem for accessing different domain that could be solved by:
      * api gateway (sync): service must hit other service thru API gateway
      * event hub/pubsub (async/push): service must subscribe other services' event to retrieve the data, which causes another consistency-related problem
      * service mesh (sync): service must hit other service thru sidecar
      * directly read a replica (async/pull)
2. Shared database 
   all or some microservice accessing the same database, this have some pros and cons:
   + simpler for the developers, since they can do JOIN and transaction
   - worst kind of performance when the bottleneck is the database especially on migration and scaling
   + no consistency problem
   
3. SAGA (sequence of local transaction)
   this have pros and cons:
   + split an atomic transaction into multiple steps that when one of the steps are failed, must reconcile/undo/create a compensation action
   - more complex than normal database transaction
4. API Composition (join inside the API service, the pattern used in Trino)
   this have pros and cons:
   + can do join across services and datasource
   - must hit multiple services (slower than normal join) if the count are large
   - can be bad if the other service calling another service too (cascading N+1 queries), eg. A hit B, B hit C, but this can be solved if B and C have batch APIs (that usually using WHERE IN, instead of single API)
5. CQRS (Command Query Responsibility Segregation)
   a pattern that created because old databases usually single master, multiple slave, but this also have a good benefit and cons:
   + simpler scaling, either need to scale the write or just scale the read
   - possible inconsistency problem if not reading from the master for transaction which adds complexity on development (which one must read the master, which one can read the readonly replica)
   
6. Domain Events 
   service must publish events, have pros and cons:
   + decoupling, no more service mesh/hitting other services, we just need to subscribe the events
   - eventual consistency
   - must store and publish events that probably never need to be consumed, but we can use directly read event database of that service to overcome this, but this can be also a benefit since events helps the auditing
7. Event Sourcing
   this pattern create a snapshot to reconstruct final state from series of events, have pros and cons:
   + can be used for reliably publish an event when state changed
   + good for auditing
   + theoritically easier to track when business logic changed (but we must build the full DFA/NFA state graph to reliably consider the edge cases)
   - difficult to query since it's series of events, unless you are prioritizing the snapshot
   
   

Engineering is about making decision and prioritization, which simplicity that needs to be prioritized, either maintainability, raw performance, ease of scaling, or other metrics, you'll need to define your own "best". And there's no silver bullet, each solution are best only for specific use case. 

But.. if you have to create a ultimate/general purpose service which patter you would use?

Based on experience, I'll use:

1. CQRS (1 writer, 1-N reader)
   reader must cache the reads
   writer must log each changes (Domain Events)
   writer can be sync or async (for slow computation, or those that depends on another service/SAGA) and update snapshot each time
2. Domain Events
   this logs of events can be tailed (async/pull) from another service if they need it
   they must record their own bookmark (of which events already tailed/consumed/ack'ed, which hasn't)
   EDIT 2021-08-16: this guy has the same idea, except that I believe there's should be no circular dependency (eg, his order-inventory system should be composed from order, inventory, and delivery)
3. API Composition
   but we can use JOIN inside the domain
   especially for statistics, we must collect each services statistics for example
   API have 2 version: consistent version (read from master), eventual consistency version (read from readonly replica)
   API must have batch/paged version other than standard CRUD
   API must tell whether that API depends on another service's APIs
4. Private database is a must
   since bottleneck are mostly always the database, it'll be better to split databases by domains from beginning (but no need to create a readonly replicata until the read part became the bottleneck)
   If the write more than 200K-600K rps i prefer manual partitioning instead of sharding (unless the database I use support sharding, automatic rebalancing, super-easy to add a new node with Tarantool-like performance)
   What if you need joins for analytics reasons? You can use Trino/Presto/BigQuery/etc, or just delegate the statistics responsibility to each services then collect/aggregate from statistics/collector service.

Go ORM Benchmark on MemSQL

I was looking for fastest ORM in Go, and found that someone already do the benchmark: kihamo's code which based on beego's (my fork for this benchmark), the result for 5 iteration was:

MemSQL 6.7.16

 10000 times - Insert
       raw:     1.54s       153927 ns/op     592 B/op     15 allocs/op
       orm:     1.60s       160195 ns/op    1465 B/op     39 allocs/op
       qbs:     1.73s       172760 ns/op    4595 B/op    107 allocs/op
      modl:     2.26s       225537 ns/op    1352 B/op     31 allocs/op
      gorp:     2.38s       238256 ns/op    1424 B/op     32 allocs/op
      xorm:     2.44s       243955 ns/op    2594 B/op     69 allocs/op
      hood:     2.75s       275120 ns/op   10812 B/op    161 allocs/op
  upper.io:     2.99s       299289 ns/op   11829 B/op    644 allocs/op
      gorm:     4.07s       407045 ns/op    7716 B/op    151 allocs/op

  2500 times - MultiInsert 100 row
       orm:     3.23s      1290337 ns/op  136250 B/op   1537 allocs/op
       raw:     3.42s      1366141 ns/op  140920 B/op    817 allocs/op
      xorm:     5.11s      2044104 ns/op  267877 B/op   4671 allocs/op
      hood:     Not support multi insert
      modl:     Not support multi insert
       qbs:     Not support multi insert
  upper.io:     Not support multi insert
      gorp:     Not support multi insert
      gorm:     Not support multi insert

 10000 times - Update
       raw:     1.43s       142733 ns/op     656 B/op     17 allocs/op
       orm:     1.50s       150464 ns/op    1424 B/op     40 allocs/op
       qbs:     1.71s       170803 ns/op    4594 B/op    107 allocs/op
      modl:     2.11s       211200 ns/op    1528 B/op     39 allocs/op
      gorp:     2.14s       213744 ns/op    1576 B/op     38 allocs/op
      hood:     2.65s       265383 ns/op   10812 B/op    161 allocs/op
      xorm:     3.01s       300524 ns/op    2697 B/op    103 allocs/op
      gorm:     8.41s       841018 ns/op   18677 B/op    389 allocs/op
  upper.io:     0.00s      0.03 ns/op       0 B/op      0 allocs/op

 20000 times - Read
       raw:     3.45s       172556 ns/op    1472 B/op     40 allocs/op
       orm:     3.81s       190347 ns/op    2649 B/op     96 allocs/op
      modl:     6.61s       330343 ns/op    1912 B/op     48 allocs/op
      gorp:     6.85s       342620 ns/op    1912 B/op     55 allocs/op
      hood:     7.02s       350974 ns/op    4098 B/op     54 allocs/op
       qbs:     7.46s       373004 ns/op    6574 B/op    175 allocs/op
  upper.io:     8.07s       403673 ns/op   10089 B/op    456 allocs/op
      gorm:     8.35s       417320 ns/op   12195 B/op    242 allocs/op
      xorm:     9.18s       459213 ns/op    9390 B/op    263 allocs/op

 10000 times - MultiRead limit 100
       raw:     6.42s       642379 ns/op   34746 B/op   1323 allocs/op
      modl:     7.59s       759230 ns/op   49902 B/op   1724 allocs/op
      gorp:     7.74s       773598 ns/op   63723 B/op   1912 allocs/op
       orm:     9.15s       914736 ns/op   85050 B/op   4286 allocs/op
       qbs:    10.16s      1016412 ns/op  165861 B/op   6429 allocs/op
  upper.io:    10.69s      1068507 ns/op   83801 B/op   2055 allocs/op
      hood:    12.43s      1243334 ns/op  136238 B/op   6364 allocs/op
      gorm:    16.23s      1622574 ns/op  254781 B/op   6229 allocs/op
      xorm:    17.39s      1738862 ns/op  180066 B/op   8093 allocs/op

The upper.io/db can't generate more efficient query than current implementation: (SET id = ? WHERE id = ?), so the update will fail on MemSQL.

Raw still the best, and Beego's built-in orm quite good except for multiread part.
Compared to MySQL with 5 iteration:

MySQL 5.7.28

 10000 times - Insert
       raw:    81.79s      8179045 ns/op     592 B/op     15 allocs/op
       qbs:    86.66s      8666472 ns/op    4595 B/op    107 allocs/op
      gorp:    88.69s      8868999 ns/op    1424 B/op     32 allocs/op
       orm:    90.29s      9028890 ns/op    1464 B/op     39 allocs/op
      hood:    91.96s      9196392 ns/op   10814 B/op    161 allocs/op
      gorm:    93.31s      9331332 ns/op    7718 B/op    151 allocs/op
      modl:    93.63s      9362930 ns/op    1352 B/op     31 allocs/op
  upper.io:    95.56s      9556491 ns/op   11830 B/op    644 allocs/op
      xorm:    96.82s      9682337 ns/op    2594 B/op     69 allocs/op

  2500 times - MultiInsert 100 row
       raw:    32.92s     13167271 ns/op  140922 B/op    818 allocs/op
       orm:    35.29s     14117094 ns/op  136296 B/op   1537 allocs/op
      xorm:    39.70s     15879522 ns/op  267943 B/op   4671 allocs/op
       qbs:     Not support multi insert
      hood:     Not support multi insert
      modl:     Not support multi insert
      gorm:     Not support multi insert
  upper.io:     Not support multi insert
      gorp:     Not support multi insert

 10000 times - Update
  upper.io:     3.08s       307724 ns/op   17735 B/op    951 allocs/op
       qbs:    87.03s      8703447 ns/op    4594 B/op    107 allocs/op
      gorp:    87.76s      8776111 ns/op    1576 B/op     38 allocs/op
      hood:    90.29s      9028560 ns/op   10813 B/op    161 allocs/op
       raw:    91.07s      9107205 ns/op     656 B/op     17 allocs/op
      modl:    92.25s      9225025 ns/op    1528 B/op     39 allocs/op
      xorm:    96.47s      9646503 ns/op    2697 B/op    103 allocs/op
      gorm:    96.90s      9690444 ns/op   18676 B/op    389 allocs/op
       orm:    99.90s      9989899 ns/op    1424 B/op     40 allocs/op

 20000 times - Read
       raw:     1.70s        84844 ns/op    1472 B/op     40 allocs/op
       orm:     1.91s        95393 ns/op    2649 B/op     96 allocs/op
       qbs:     1.92s        96013 ns/op    6576 B/op    175 allocs/op
      hood:     2.89s       144473 ns/op    4097 B/op     54 allocs/op
      gorp:     2.95s       147612 ns/op    1912 B/op     55 allocs/op
      modl:     2.99s       149255 ns/op    1912 B/op     48 allocs/op
  upper.io:     4.33s       216621 ns/op   10089 B/op    456 allocs/op
      gorm:     4.35s       217446 ns/op   12195 B/op    242 allocs/op
      xorm:     4.68s       234212 ns/op    9392 B/op    263 allocs/op

 10000 times - MultiRead limit 100
       raw:     3.48s       348355 ns/op   34744 B/op   1323 allocs/op
      modl:     4.56s       455775 ns/op   49904 B/op   1724 allocs/op
      gorp:     4.94s       494206 ns/op   63725 B/op   1912 allocs/op
       orm:     5.97s       597024 ns/op   85060 B/op   4286 allocs/op
  upper.io:     6.64s       664491 ns/op   83803 B/op   2055 allocs/op
       qbs:     7.29s       729417 ns/op  165864 B/op   6429 allocs/op
      hood:     8.32s       831645 ns/op  136237 B/op   6364 allocs/op
      gorm:    11.53s      1152701 ns/op  254774 B/op   6228 allocs/op
      xorm:    12.97s      1296585 ns/op  180067 B/op   8093 allocs/op

The overhead for mysql is not significant. Note that the insert and update is slow because the transaction isolation is not set to read committed.

MemSQL 7.0.9

 10000 times - Insert
       raw:     1.58s       158308 ns/op     592 B/op     15 allocs/op
       orm:     1.67s       166718 ns/op    1464 B/op     39 allocs/op
       qbs:     1.87s       186627 ns/op    4595 B/op    107 allocs/op
      modl:     2.29s       228827 ns/op    1352 B/op     31 allocs/op
      gorp:     2.45s       244721 ns/op    1424 B/op     32 allocs/op
      xorm:     2.56s       255536 ns/op    2595 B/op     69 allocs/op
      hood:     2.72s       271565 ns/op   10814 B/op    161 allocs/op
  upper.io:     3.00s       300482 ns/op   11828 B/op    644 allocs/op
      gorm:     4.15s       414676 ns/op    7717 B/op    151 allocs/op

  2500 times - MultiInsert 100 row
       orm:     3.27s      1306549 ns/op  136254 B/op   1537 allocs/op
       raw:     3.31s      1324971 ns/op  140920 B/op    817 allocs/op
      xorm:     5.19s      2077746 ns/op  267822 B/op   4671 allocs/op
      modl:     Not support multi insert
      gorm:     Not support multi insert
      hood:     Not support multi insert
  upper.io:     Not support multi insert
       qbs:     Not support multi insert
      gorp:     Not support multi insert

 10000 times - Update
       raw:     1.57s       156799 ns/op     656 B/op     17 allocs/op
       orm:     1.62s       161919 ns/op    1425 B/op     40 allocs/op
       qbs:     1.76s       176142 ns/op    4595 B/op    107 allocs/op
      modl:     2.23s       222540 ns/op    1528 B/op     39 allocs/op
      gorp:     2.29s       228606 ns/op    1576 B/op     38 allocs/op
      hood:     2.67s       266824 ns/op   10813 B/op    161 allocs/op
      xorm:     3.29s       329236 ns/op    2697 B/op    103 allocs/op
      gorm:     8.83s       882594 ns/op   18677 B/op    389 allocs/op
  upper.io:     0.00s      0.04 ns/op       0 B/op      0 allocs/op

 20000 times - Read
       raw:     3.74s       186956 ns/op    1472 B/op     40 allocs/op
       orm:     3.88s       194016 ns/op    2649 B/op     96 allocs/op
      modl:     6.51s       325522 ns/op    1912 B/op     48 allocs/op
      gorp:     6.83s       341292 ns/op    1912 B/op     55 allocs/op
       qbs:     7.35s       367283 ns/op    6574 B/op    175 allocs/op
      hood:     7.73s       386417 ns/op    4098 B/op     54 allocs/op
  upper.io:     8.76s       438185 ns/op   10089 B/op    456 allocs/op
      gorm:     9.33s       466715 ns/op   12194 B/op    242 allocs/op
      xorm:     9.89s       494368 ns/op    9390 B/op    263 allocs/op

 10000 times - MultiRead limit 100
       raw:     6.43s       642713 ns/op   34746 B/op   1323 allocs/op
      modl:     7.49s       749218 ns/op   49902 B/op   1724 allocs/op
      gorp:     7.63s       763255 ns/op   63728 B/op   1912 allocs/op
       orm:     8.95s       895022 ns/op   85050 B/op   4286 allocs/op
       qbs:    10.23s      1023162 ns/op  165861 B/op   6429 allocs/op
  upper.io:    11.28s      1127575 ns/op   83801 B/op   2055 allocs/op
      hood:    12.62s      1262190 ns/op  136241 B/op   6364 allocs/op
      gorm:    16.65s      1665189 ns/op  254772 B/op   6228 allocs/op
      xorm:    17.69s      1768666 ns/op  180053 B/op   8093 allocs/op

There's seems no significant performance difference between MemSQL 6.7 and 7.0 in this case. But what if we put MemSQL inside docker, how much the overhead?

MemSQL 7.0.9 inside docker with NAT
  docker run -i --init --name memsql1 -e LICENSE_KEY=$LICENSE_KEY -p 3306:3306 -p 8082:8080 memsql/cluster-in-a-box

 10000 times - Insert
       raw:     2.29s       228825 ns/op     592 B/op     15 allocs/op
       orm:     2.39s       238694 ns/op    1465 B/op     39 allocs/op
       qbs:     2.58s       258331 ns/op    4595 B/op    107 allocs/op
      modl:     3.60s       360296 ns/op    1352 B/op     31 allocs/op
      xorm:     3.76s       376043 ns/op    2594 B/op     69 allocs/op
      gorp:     3.77s       377271 ns/op    1424 B/op     32 allocs/op
      hood:     4.21s       421357 ns/op   10813 B/op    161 allocs/op
  upper.io:     4.41s       441370 ns/op   11829 B/op    644 allocs/op
      gorm:     6.68s       668315 ns/op    7717 B/op    151 allocs/op

  2500 times - MultiInsert 100 row
       orm:     4.38s      1750560 ns/op  136321 B/op   1537 allocs/op
       raw:     4.73s      1893901 ns/op  140920 B/op    817 allocs/op
      xorm:     6.34s      2537707 ns/op  267921 B/op   4671 allocs/op
      gorp:     Not support multi insert
      modl:     Not support multi insert
       qbs:     Not support multi insert
      gorm:     Not support multi insert
      hood:     Not support multi insert
  upper.io:     Not support multi insert

 10000 times - Update
       raw:     2.28s       228252 ns/op     656 B/op     17 allocs/op
       orm:     2.37s       237145 ns/op    1424 B/op     40 allocs/op
       qbs:     2.45s       244695 ns/op    4594 B/op    107 allocs/op
      modl:     3.52s       351694 ns/op    1528 B/op     39 allocs/op
      gorp:     3.55s       354756 ns/op    1576 B/op     38 allocs/op
      hood:     4.08s       407915 ns/op   10812 B/op    161 allocs/op
      xorm:     4.86s       486246 ns/op    2697 B/op    103 allocs/op
      gorm:    13.55s      1354767 ns/op   18679 B/op    389 allocs/op
  upper.io:     0.00s      0.03 ns/op       0 B/op      0 allocs/op

 20000 times - Read
       raw:     5.37s       268278 ns/op    1472 B/op     40 allocs/op
       orm:     5.40s       269883 ns/op    2649 B/op     96 allocs/op
       qbs:    10.20s       509797 ns/op    6574 B/op    175 allocs/op
      modl:    11.03s       551638 ns/op    1912 B/op     48 allocs/op
      gorp:    11.49s       574716 ns/op    1912 B/op     55 allocs/op
      hood:    11.76s       587919 ns/op    4097 B/op     54 allocs/op
  upper.io:    13.29s       664267 ns/op   10089 B/op    456 allocs/op
      gorm:    13.60s       679870 ns/op   12194 B/op    242 allocs/op
      xorm:    14.83s       741376 ns/op    9390 B/op    263 allocs/op

 10000 times - MultiRead limit 100
       raw:     8.34s       833549 ns/op   34747 B/op   1323 allocs/op
      modl:     9.73s       972505 ns/op   49902 B/op   1724 allocs/op
      gorp:     9.95s       994607 ns/op   63725 B/op   1912 allocs/op
       orm:    11.24s      1123517 ns/op   85058 B/op   4286 allocs/op
       qbs:    12.12s      1212164 ns/op  165860 B/op   6429 allocs/op
  upper.io:    13.96s      1396187 ns/op   83800 B/op   2055 allocs/op
      hood:    16.05s      1604510 ns/op  136241 B/op   6364 allocs/op
      gorm:    20.23s      2023026 ns/op  254764 B/op   6228 allocs/op
      xorm:    20.45s      2044591 ns/op  180065 B/op   8093 allocs/op

It shown that running MemSQL inside docker has about ~44% performance penalty, it seems to be the NAT bottleneck. Let's try again using host network:

MemSQL 7.0.9 inside docker with host network

 docker run -i --init --name memsql1 -e LICENSE_KEY=$LICENSE_KEY --net=host memsql/cluster-in-a-box

 10000 times - Insert
       raw:     1.84s       184249 ns/op     592 B/op     15 allocs/op
       orm:     1.98s       197552 ns/op    1465 B/op     39 allocs/op
       qbs:     2.02s       201505 ns/op    4595 B/op    107 allocs/op
      gorp:     2.62s       262421 ns/op    1424 B/op     32 allocs/op
      modl:     2.63s       263243 ns/op    1352 B/op     31 allocs/op
      xorm:     2.87s       287027 ns/op    2594 B/op     69 allocs/op
      hood:     3.18s       317792 ns/op   10814 B/op    161 allocs/op
  upper.io:     3.50s       350001 ns/op   11828 B/op    644 allocs/op
      gorm:     4.69s       469475 ns/op    7716 B/op    151 allocs/op

  2500 times - MultiInsert 100 row
       orm:     4.02s      1606318 ns/op  136207 B/op   1537 allocs/op
       raw:     4.26s      1702967 ns/op  140921 B/op    818 allocs/op
      xorm:     6.16s      2463782 ns/op  267869 B/op   4671 allocs/op
      gorp:     Not support multi insert
      gorm:     Not support multi insert
       qbs:     Not support multi insert
      modl:     Not support multi insert
  upper.io:     Not support multi insert
      hood:     Not support multi insert

 10000 times - Update
       raw:     1.85s       184579 ns/op     656 B/op     17 allocs/op
       orm:     1.97s       197037 ns/op    1425 B/op     40 allocs/op
       qbs:     1.97s       197209 ns/op    4595 B/op    107 allocs/op
      modl:     2.60s       259853 ns/op    1528 B/op     39 allocs/op
      gorp:     2.61s       260791 ns/op    1576 B/op     38 allocs/op
      hood:     3.11s       311218 ns/op   10814 B/op    161 allocs/op
      xorm:     3.75s       374953 ns/op    2697 B/op    103 allocs/op
      gorm:    10.18s      1017593 ns/op   18676 B/op    389 allocs/op
  upper.io:     0.00s      0.04 ns/op       0 B/op      0 allocs/op

 20000 times - Read
       raw:     4.34s       217164 ns/op    1472 B/op     40 allocs/op
       orm:     4.51s       225554 ns/op    2649 B/op     96 allocs/op
      gorp:     8.39s       419645 ns/op    1912 B/op     55 allocs/op
       qbs:     8.79s       439281 ns/op    6574 B/op    175 allocs/op
      hood:     8.97s       448493 ns/op    4098 B/op     54 allocs/op
      modl:     9.14s       456942 ns/op    1912 B/op     48 allocs/op
  upper.io:    10.57s       528673 ns/op   10089 B/op    456 allocs/op
      gorm:    11.07s       553741 ns/op   12194 B/op    242 allocs/op
      xorm:    11.93s       596566 ns/op    9391 B/op    263 allocs/op

 10000 times - MultiRead limit 100
       raw:     7.92s       792363 ns/op   34747 B/op   1323 allocs/op
      modl:     9.13s       912642 ns/op   49902 B/op   1724 allocs/op
      gorp:     9.35s       934646 ns/op   63722 B/op   1912 allocs/op
       orm:    10.35s      1035154 ns/op   85049 B/op   4286 allocs/op
       qbs:    11.41s      1141194 ns/op  165860 B/op   6429 allocs/op
  upper.io:    13.05s      1304687 ns/op   83800 B/op   2055 allocs/op
      hood:    14.64s      1463870 ns/op  136245 B/op   6364 allocs/op
      gorm:    18.76s      1876366 ns/op  254767 B/op   6228 allocs/op
      xorm:    20.16s      2015870 ns/op  180055 B/op   8093 allocs/op

This version only 14-16% slower than baremetal version. There's also another alternative, using iptables forwarding, :

MemSQL 7.0.9 inside docker with iptables forwarding

  docker run -i --init --name memsql1 -e LICENSE_KEY=$LICENSE_KEY memsql/cluster-in-a-box
  sudo sysctl -w net.ipv4.conf.all.route_localnet=1
  GUEST_IP=$(docker inspect --format '{{ .NetworkSettings.IPAddress }}' memsql1)
  sudo iptables -t nat -A OUTPUT -m addrtype --src-type LOCAL --dst-type LOCAL -p tcp --dport 3306 -j DNAT --to-destination $GUEST_IP
  sudo iptables -t nat -A POSTROUTING -m addrtype --src-type LOCAL --dst-type UNICAST -j MASQUERADE
  # replace -A with -D to delete the rule after using

 10000 times - Insert
       raw:     1.94s       193731 ns/op     592 B/op     15 allocs/op

  2500 times - MultiInsert 100 row
       raw:     4.53s      1813220 ns/op  140922 B/op    818 allocs/op

 10000 times - Update
       raw:     1.90s       190419 ns/op     656 B/op     17 allocs/op

 20000 times - Read
       raw:     4.45s       222618 ns/op    1472 B/op     40 allocs/op

 10000 times - MultiRead limit 100
       raw:     7.64s       764215 ns/op   34746 B/op   1323 allocs/op

Which is 19-23.5% slower. Another alternative is using gost (or another proxy like socat, ncat, goproxy, redir, etc):

MemSQL 7.0.9 inside docker with gost proxy

  docker run -i --init --name memsql1 -e LICENSE_KEY=$LICENSE_KEY memsql/cluster-in-a-box
  gost -L tcp://:3306/$GUEST_IP:3306

 10000 times - Insert
       raw:     2.32s       231583 ns/op     592 B/op     15 allocs/op

  2500 times - MultiInsert 100 row
       raw:     4.68s      1870160 ns/op  140921 B/op    818 allocs/op

 10000 times - Update
       raw:     2.25s       224826 ns/op     656 B/op     17 allocs/op

 20000 times - Read
       raw:     5.31s       265496 ns/op    1472 B/op     40 allocs/op

 10000 times - MultiRead limit 100
       raw:     8.10s       809700 ns/op   34747 B/op   1323 allocs/op

Which apparently 41-47% slower, as bad as docker's NAT. Now, what if we access the IP directly?

MemSQL 7.0.9 inside docker direct ip access

  docker run -i --init --name memsql1 -e LICENSE_KEY=$LICENSE_KEY memsql/cluster-in-a-box
 orm-benchmark -multi=5 -orm=raw -source "root:@tcp($GUEST_IP:3306)/orm_bench?charset=utf8"

 10000 times - Insert
       raw:     1.76s       176259 ns/op     592 B/op     15 allocs/op

  2500 times - MultiInsert 100 row
       raw:     4.19s      1675736 ns/op  140922 B/op    818 allocs/op

 10000 times - Update
       raw:     1.72s       171562 ns/op     656 B/op     17 allocs/op

 20000 times - Read
       raw:     4.06s       202945 ns/op    1472 B/op     40 allocs/op

 10000 times - MultiRead limit 100
       raw:     7.50s       750091 ns/op   34746 B/op   1323 allocs/op

This aproach only have 8-12% overhead. If I have more free time, I'll benchmark volume binding performance. Or maybe someone else want to contribute adding more ORMs? (see TODO section on github repo)

Huge List of Database Benchmark

Today we will benchmark a single node version of distributed database (and some non-distributed database for comparison), the client all written with Go (with any available driver). The judgement will be about performance (that mostly write, and some infrequent read), not about the distribution performance (I will take a look in some other time). I searched a lot of database from DbEngines for database that could suit my needs for my next project. For session kv-store I'll be using obviously first choice is Aerospike, but since they cannot be run inside server that I rent (that uses OpenVZ), so I'll go for second choice that is Redis. Here's the list of today's contender:

• CrateDB, a highly optimized for huge amount of data (they said), probably would be the best for updatable time series, also with built-in search engine, so this one is quite fit my use case probably to replace [Riot (small scale) or Manticore (large scale)] and [InfluxDB or TimescaleDB], does not support auto increment
• CockroachDB, self-healing database with PostgreSQL-compatible connector, the community edition does not support table partitioning
• MemSQL, which also can replace kv-store, there's a limit of 128GB RAM for free version. Row-store tables can only have one PRIMARY key or one UNIQUE key or one AUTO increment column that must be a SHARD key, and it cannot be updated or altered. Column-store tables does not support UNIQUE/PRIMARY key, only SHARD KEY. The client/connector is MySQL-compatible
• MariaDB (MySQL), one of the most popular open source RDBMS, for the sake of comparison
• PostgreSQL, my favorite RDBMS, for the sake of comparison 
• NuoDB on another benchmark even faster than GoogleSpanner or CockroachDB, the community edition only support 3 transaction engine (TE) and 1 storage manager (SM)
• YugaByteDB, distributed KV+SQL with Cassandra and PostgreSQL compatible protocol.  Some of SQL syntax not yet supported (ALTER USER, UNIQUE on CREATE TABLE).
• ScyllaDB, a C++ version of Cassandra. All Cassandra-like databases has a lot of restrictions/annoyances by design compared to traditional RDBMS (cannot CREATE INDEX on composite PRIMARY KEY, no AUTO INCREMENT, doesn't support UNION ALL or OR operator, must use COUNTER type if you want to UPDATE x=x+n, cannot mix COUNTER type with non-counter type on the same table, etc), does not support ORDER BY other than clustering key, does not support OFFSET on LIMIT.
• Clickhouse, claimed to be fastest and one of the most storage space efficient OLAP database, but doesn't support UPDATE/DELETE-syntax (requires ALTER TABLE to UPDATE/DELETE), only support batch insert, does not support UNIQUE, AUTO INCREMENT. Since this is not designed to be an OLTP database, obviously this benchmark would be totally unfair for Clickhouse.

What's the extra motivation of this post?

I almost never use distributed database, since all of my project have no more than 200 concurrent users/sec. I've encountered bottleneck before, and the culprit is multiple slow complex queries, that could be solved by queuing to another message queue, and process them one by one instead of bombing database's process at the same time and hogging out the memory.

The benchmark scenario would be like this:
1. 50k inserts of single column string value, 200k inserts of 2 column unique value, 900k insert of unique
INSERT INTO users(id, uniq_str) -- x50k
INSERT INTO items(fk_id, typ, amount) -- x50k x4
INSERT INTO rels(fk_low, fk_high, bond) -- x900k

2. while inserting at 5%+, there would be at least 100k random search queries of unique value/, and 300k random search queries, every search queries, there would be 3 random update of amount
SELECT * FROM users WHERE uniq_str = ? -- x100k
SELECT * FROM items WHERE fk_id = ? AND typ IN (?) -- x100k x3
UPDATE items SET amount = amount + xxx WHERE id = ? -- x100k x3

3. while inserting at 5%+, there would be also at least 100k random search queries
SELECT * FROM items WHERE fk_id = ?

4. while inserting at 5%+, there also at least 200k query of relations and 50% chance to update the bond
SELECT * FROM rels WHERE fk_low = ? or fk_high = ? -- x200k
UPDATE rels SET bond = bond + xxx WHERE id = ? -- x200k / 2

This benchmark represent simplified real use case of the game I'm currently develop. Let's start with PostgreSQL 10.7 (current one on Ubuntu 18.04.1 LTS), the configuration generated by pgtune website:

max_connections = 400
shared_buffers = 8GB
effective_cache_size = 24GB
maintenance_work_mem = 2GB
checkpoint_completion_target = 0.9
wal_buffers = 16MB
default_statistics_target = 100
random_page_cost = 1.1
effective_io_concurrency = 200
work_mem = 5242kB
min_wal_size = 2GB
max_wal_size = 4GB
max_worker_processes = 8
max_parallel_workers_per_gather = 4
max_parallel_workers = 8

Create the user and database first:

sudo su - postgres
createuser b1
createdb b1
psql 
GRANT ALL PRIVILEGES ON DATABASE b1 TO b1
\q

Add to pg_hba.conf if required, then restart:

local   all b1 trust
host all b1 127.0.0.1/32 trust
host all b1 ::1/128 trust

For slow databases, all values reduced by 20 except query-only.

$ go run pg.go lib.go

[Pg] RandomSearchItems (100000, 100%) took 24.62s (246.21 µs/op)

[Pg] SearchRelsAddBonds (10000, 100%) took 63.73s (6372.56 µs/op)
[Pg] UpdateItemsAmounts (5000, 100%) took 105.10s (21019.88 µs/op)
[Pg] InsertUsersItems (2500, 100%) took 129.41s (51764.04 µs/op)
USERS CR    :    2500 /    4999 
ITEMS CRU   :   17500 /   14997 +  698341 / 14997
RELS  CRU   :    2375 /   16107 / 8053
SLOW FACTOR : 20
CRU µs/rec  : 5783.69 / 35.26 / 7460.65

Next we'll try with MySQL 5.7, create user and database first, then multiply all memory config by 10 (since there are automatic config generator for mysql?):

innodb_buffer_pool_size=4G

$ sudo mysql

CREATE USER 'b1'@'localhost' IDENTIFIED BY 'b1';

CREATE DATABASE b1;

GRANT ALL PRIVILEGES ON b1.* TO 'b1'@'localhost';

FLUSH PRIVILEGES;

sudo mysqltuner # not sure if this useful

And here's the result:

$ go run maria.go lib.go

[My] RandomSearchItems (100000, 100%) took 16.62s (166.20 µs/op)
[My] SearchRelsAddBonds (10000, 100%) took 86.32s (8631.74 µs/op)
[My] UpdateItemsAmounts (5000, 100%) took 172.35s (34470.72 µs/op)
[My] InsertUsersItems (2500, 100%) took 228.52s (91408.86 µs/op)
USERS CR    :    2500 /    4994 
ITEMS CRU   :   17500 /   14982 +  696542 / 13485 
RELS  CRU   :    2375 /   12871 / 6435 
SLOW FACTOR : 20 
CRU µs/rec  : 10213.28 / 23.86 / 13097.44

Next we'll try with MemSQL 6.7.16-55671ba478, while the insert and update performance is amazing, the query/read performance is 3-4x slower than traditional RDBMS:

$ memsql-admin start-node --all

$ go run memsql.go lib.go # 4 sec before start RU

[Mem] InsertUsersItems (2500, 100%) took 4.80s (1921.97 µs/op)
[Mem] UpdateItemsAmounts (5000, 100%) took 13.48s (2695.83 µs/op)
[Mem] SearchRelsAddBonds (10000, 100%) took 14.40s (1440.29 µs/op)
[Mem] RandomSearchItems (100000, 100%) took 64.87s (648.73 µs/op)
USERS CR    :    2500 /    4997 
ITEMS CRU   :   17500 /   14991 +  699783 / 13504 
RELS  CRU   :    2375 /   19030 / 9515 
SLOW FACTOR : 20 
CRU µs/rec  : 214.75 / 92.70 / 1255.93

$ go run memsql.go lib.go # 2 sec before start RU

[Mem] InsertUsersItems (2500, 100%) took 5.90s (2360.01 µs/op)

[Mem] UpdateItemsAmounts (5000, 100%) took 13.76s (2751.67 µs/op)

[Mem] SearchRelsAddBonds (10000, 100%) took 14.56s (1455.95 µs/op)

[Mem] RandomSearchItems (100000, 100%) took 65.30s (653.05 µs/op)

USERS CR    :    2500 /    4998 

ITEMS CRU   :   17500 /   14994 +  699776 / 13517 

RELS  CRU   :    2375 /   18824 / 9412 

SLOW FACTOR : 20 

CRU µs/rec  : 263.69 / 93.32 / 1282.38

$ go run memsql.go lib.go # SLOW FACTOR 5

[Mem] InsertUsersItems (10000, 100%) took 31.22s (3121.90 µs/op)

[Mem] UpdateItemsAmounts (20000, 100%) took 66.55s (3327.43 µs/op)

[Mem] RandomSearchItems (100000, 100%) took 85.13s (851.33 µs/op)

[Mem] SearchRelsAddBonds (40000, 100%) took 133.05s (3326.29 µs/op)
USERS CR    :   10000 /   19998
ITEMS CRU   :   70000 /   59994 +  699944 / 53946

RELS  CRU   :   37896 /  300783 / 150391

SLOW FACTOR : 5

CRU µs/rec  : 264.80 / 121.63 / 1059.16

$ go run memsql.go lib.go # SLOW FACTOR 1, DB SIZE: 548.2 MB
[Mem] RandomSearchItems (100000, 100%) took 88.84s (888.39 µs/op)
[Mem] UpdateItemsAmounts (100000, 100%) took 391.87s (3918.74 µs/op)
[Mem] InsertUsersItems (50000, 100%) took 482.57s (9651.42 µs/op)
[Mem] SearchRelsAddBonds (200000, 100%) took 5894.22s (29471.09 µs/op)
USERS CR    :   50000 /   99991 
ITEMS CRU   :  350000 /  299973 +  699846 / 269862 
RELS  CRU   :  946350 / 7161314 / 3580657 
SLOW FACTOR : 1
CRU µs/rec  : 358.43 / 126.94 / 1549.13

Column store tables with MemSQL 6.7.16-55671ba478:

$ go run memsql-columnstore.go lib.go # SLOW FACTOR 20
[Mem] InsertUsersItems (2500, 100%) took 6.44s (2575.26 µs/op)
[Mem] UpdateItemsAmounts (5000, 100%) took 17.51s (3502.94 µs/op)
[Mem] SearchRelsAddBonds (10000, 100%) took 18.82s (1881.71 µs/op)
[Mem] RandomSearchItems (100000, 100%) took 79.48s (794.78 µs/op)
USERS CR    :    2500 /    4997 
ITEMS CRU   :   17500 /   14991 +  699776 / 13512 
RELS  CRU   :    2375 /   18861 / 9430 
SLOW FACTOR : 20 
CRU µs/rec  : 287.74 / 113.58 / 1645.84

Next we'll try CrateDB 3.2.7, with similar setup like PostgreSQL, the result:

$ go run crate.go lib.go

[Crate] SearchRelsAddBonds (10000, 100%) took 49.11s (4911.38 µs/op)
[Crate] RandomSearchItems (100000, 100%) took 101.40s (1013.95 µs/op)
[Crate] UpdateItemsAmounts (5000, 100%) took 246.42s (49283.84 µs/op)
[Crate] InsertUsersItems (2500, 100%) took 306.12s (122449.00 µs/op)
USERS CR    :    2500 /    4965 
ITEMS CRU   :   17500 /   14894 +  690161 / 14895 
RELS  CRU   :    2375 /    4336 / 2168 
SLOW FACTOR : 20 
CRU µs/rec  : 13681.45 / 146.92 / 19598.85

Next is CockroachDB 19.1, the result:

$ go run cockroach.go lib.go
[Cockroach] SearchRelsAddBonds (10000, 100%) took 59.25s (5925.42 µs/op)
[Cockroach] RandomSearchItems (100000, 100%) took 85.84s (858.45 µs/op)
[Cockroach] UpdateItemsAmounts (5000, 100%) took 261.43s (52285.65 µs/op
[Cockroach] InsertUsersItems (2500, 100%) took 424.66s (169864.55 µs/op)
USERS CR    :    2500 /    4988
ITEMS CRU   :   17500 /   14964 +  699331 / 14964 
RELS  CRU   :    2375 /    5761 / 2880 
SLOW FACTOR : 20 
CRU µs/rec  : 18979.28 / 122.75 / 19022.43

Next is NuoDB 3.4.1, the storage manager and transaction engine config and the benchmark result:

$ chown nuodb:nuodb /media/nuodb
$ nuodbmgr --broker localhost --password nuodb1pass
  start process sm archive /media/nuodb host localhost database b1 initialize true 
  start process te host localhost database b1 
    --dba-user b2 --dba-password b3
$ nuosql b1 --user b2 --password b3

$ go run nuodb.go lib.go
[Nuo] RandomSearchItems (100000, 100%) took 33.79s (337.90 µs/op)
[Nuo] SearchRelsAddBonds (10000, 100%) took 72.18s (7218.04 µs/op)
[Nuo] UpdateItemsAmounts (5000, 100%) took 117.22s (23443.65 µs/op)
[Nuo] InsertUsersItems (2500, 100%) took 144.51s (57804.21 µs/op)
USERS CR    :    2500 /    4995 
ITEMS CRU   :   17500 /   14985 +  698313 / 14985 
RELS  CRU   :    2375 /   15822 / 7911 
SLOW FACTOR : 20 
CRU µs/rec  : 6458.57 / 48.39 / 8473.22

Next is TiDB 2.1.7, the config and the result:

sudo sysctl -w net.core.somaxconn=32768
sudo sysctl -w vm.swappiness=0
sudo sysctl -w net.ipv4.tcp_syncookies=0
sudo sysctl -w fs.file-max=1000000

$ pd-server --name=pd1 \
                --data-dir=pd1 \
                --client-urls="http://127.0.0.1:2379" \
                --peer-urls="http://127.0.0.1:2380" \
                --initial-cluster="pd1=http://127.0.0.1:2380" \
                --log-file=pd1.log
$ tikv-server --pd-endpoints="127.0.0.1:2379" \
                --addr="127.0.0.1:20160" \
                --data-dir=tikv1 \
                --log-file=tikv1.log
$ tidb-server --store=tikv --path="127.0.0.1:2379" --log-file=tidb.log

$ go run tidb.go lib.go
[Ti] InsertUsersItems (125, 5%) took 17.59s (140738.00 µs/op)
[Ti] SearchRelsAddBonds (500, 5%) took 9.17s (18331.36 µs/op)
[Ti] RandomSearchItems (5000, 5%) took 10.82s (2163.28 µs/op)
# failed with bunch of errors on tikv, such as:
[2019/04/26 04:20:11.630 +07:00] [ERROR] [endpoint.rs:452] [error-response] [err="locked LockInfo { primary_lock: [116, 128, 0, 0, 0, 0, 0, 0, 50, 95, 114, 128, 0, 0, 0, 0, 0, 0, 96], lock_version: 407955626145349685, key: [116, 128, 0, 0, 0, 0, 0, 0, 50, 95, 114, 128, 0, 0, 0, 0, 0, 0, 96], lock_ttl: 3000, unknown_fields: UnknownFields { fields: None }, cached_size: CachedSize { size: 0 } }"]

Next is YugaByte 1.2.5.0, the result:

export YB_PG_FALLBACK_SYSTEM_USER_NAME=user1
./bin/yb-ctl --data_dir=/media/yuga create
# edit yb-ctl set use_cassandra_authentication = True
./bin/yb-ctl --data_dir=/media/yuga start

./bin/cqlsh -u cassandra -p cassandra

./bin/psql -h 127.0.0.1 -p 5433 -U postgres
CREATE DATABASE b1;

GRANT ALL ON b1 TO postgres;

$ go run yuga.go lib.go
[Yuga] InsertUsersItems (2500, 100%) took 116.42s (46568.71 µs/op)
[Yuga] UpdateItemsAmounts (5000, 100%) took 173.10s (34620.48 µs/op)
[Yuga] RandomSearchItems (100000, 100%) took 350.04s (3500.43 µs/op)
[Yuga] SearchRelsAddBonds (10000, 100%) took 615.17s (61516.91 µs/op)
USERS CR    :    2500 /    4999 

ITEMS CRU   :   17500 /   14997 +  699587 / 14997 

RELS  CRU   :    2375 /   18713 / 9356 

SLOW FACTOR : 20 

CRU µs/rec  : 5203.21 / 500.36 / 38646.88

Next is ScyllaDB 3.0.8, the result:

$ cqlsh
CREATE KEYSPACE b1 WITH replication = {'class': 'SimpleStrategy', 'replication_factor' : 1};

$ go run scylla.go lib.go
[Scylla] InsertUsersItems (2500, 100%) took 10.92s (4367.99 µs/op)
[Scylla] UpdateItemsAmounts (5000, 100%) took 26.85s (5369.63 µs/op)
[Scylla] SearchRelsAddBonds (10000, 100%) took 28.70s (2870.26 µs/op)
[Scylla] RandomSearchItems (100000, 100%) took 49.74s (497.41 µs/op)
USERS CR    :    2500 /    5000 
ITEMS CRU   :   17500 /   14997 +  699727 / 15000 
RELS  CRU   :    2375 /    9198 / 9198 
SLOW FACTOR : 20 
CRU µs/rec  : 488.04 / 71.09 / 2455.20

Next is Clickhouse 19.7.3.9 with batch INSERT, the result:

$ go run clickhouse.go lib.go
[Click] InsertUsersItems (2500, 100%) took 13.54s (5415.17 µs/op)
[Click] RandomSearchItems (100000, 100%) took 224.58s (2245.81 µs/op)
[Click] SearchRelsAddBonds (10000, 100%) took 421.16s (42115.93 µs/op)
[Click] UpdateItemsAmounts (5000, 100%) took 581.63s (116325.46 µs/op)
USERS CR    :    2500 /    4999 
ITEMS CRU   :   17500 /   14997 +  699748 / 15000 
RELS  CRU   :    2375 /   19052 / 9526 
SLOW FACTOR : 20 
CRU µs/rec  : 605.05 / 320.95 / 41493.35

When INSERT is not batched on Clickhouse 19.7.3.9:

$ go run clickhouse-1insertPreTransaction.go lib.go
[Click] InsertUsersItems (2500, 100%) took 110.78s (44312.56 µs/op)
[Click] RandomSearchItems (100000, 100%) took 306.10s (3060.95 µs/op)
[Click] SearchRelsAddBonds (10000, 100%) took 534.91s (53491.35 µs/op)
[Click] UpdateItemsAmounts (5000, 100%) took 710.39s (142078.55 µs/op)
USERS CR    :    2500 /    4999 
ITEMS CRU   :   17500 /   14997 +  699615 / 15000 
RELS  CRU   :    2375 /   18811 / 9405 
SLOW FACTOR : 20 
CRU µs/rec  : 4951.12 / 437.52 / 52117.48

These benchmark performed using i7-4720HQ 32GB RAM with SSD disk. At first there's a lot that I want to add to this benchmark (maybe someday) to make this huge '__'), such as:

• DGraph, a graph database written in Go, the backup is local (same as MemSQL, so you cannot do something like this ssh foo@bar "pg_dump | xz - -c" | pv -r -b > /tmp/backup_`date +%Y%m%d_%H%M%S`.sql.xz")
• Cayley, a graph layer written in Go, can support many backend storage
• ArangoDB, multi-model database, with built-in Foxx Framework for creating REST APIs, has unfamiliar AQL syntax
• MongoDB, one of the most popular open source document database, for the sake of comparison, I'm not prefer this one because of the memory usage.
• InfluxDB or TimeScaleDB or SiriDB or GridDB for comparison with Clickhouse
• Redis or SSDB or LedisDB or Codis or Olric or SummitDB, obviously for the sake of comparison. Also Cete, distributed key-value but instead using memcache protocol this one uses gRPC and REST
• Tarantool, a redis competitor with ArrangoDB-like features but with Lua instead of JS, I want to see if this simpler to use but with near equal performance as Aerospike
• Aerospike, fastest distributed kv-store I ever test, just for the sake of comparison, the free version limited to 2 namespace with 4 billions object. Too bad this one cannot be started on OpenVZ-based VM.
• Couchbase, document oriented database that support SQL-like syntax (N1QL), the free for production one is few months behind the enterprise edition. Community edition cannot create index (always error 5000?).
• GridDB, in-memory database from Toshiba, benchmarked to be superior to Cassandra
• ClustrixDB (New name: MariaDB XPand), distributed columnstore version of MariaDB, community version does not support automatic failover and non-blocking backup
• Altibase, open source in-memory database promoted to be Oracle-compatible, not sure what's the limitation of the open source version.
• RedisGraph, fastest in-memory graph database, community edition available.

Skipped databases:

• RethinkDB, document-oriented database, last ubuntu package cannot be installed, probably because the project no longer maintained
• OrientDB, multi model (document and graph database), their screenshot looks interesting, multi-model graph database, but too bad both Golang driver are unmaintained and probably unusable for latest version (3.x)
• TiDB, a work in progress approach of CockroachDB but with MySQL-compatible connector, as seen from benchmark above, there's a lot of errors happening
• RQLite, a distributed SQLite, the go driver by default not threadsafe
• VoltDB, seems not free, since the website shows "free evaluation"
• HyperDex, have good benchmark on paper, but no longer maintained
• LMDB-memcachedb, faster version of memcachedb, a distributed kv, but no longer maintained
• FoundationDB, a multi-model database, built from kv-database with additional layers for other models, seems to have complicated APIs
• TigerGraph, fastest distributed graph database, developer edition free but may not be used for production

For now, I have found what I need, so probably i'll add the rest later. The code for this benchmark can be found here: https://github.com/kokizzu/hugedbbench (send pull request then i'll run and update this post) and the spreadsheet here: http://tiny.cc/hugedb

The chart (lower is better) shown below:


Other 2018's benchmark here (tl;dr: CockroachDB mostly higher throughput, YugabyteDB lowest latency, TiDB lowest performance among those 3).