Showing posts with label aerospike. Show all posts
Showing posts with label aerospike. Show all posts

2022-04-01

Georeplicable Architecture Tech Stacks

Since I got a use case for geo-replicable (multiple datacenters around the world), I need to create metrics and billing and log aggregator that survives even when multiple datacenter is down. So we need to create an architecture that the each cluster of databases in datacenter can give correct answer on their own, the layers are:

1. Cache Layer

2. Source of Immediate Truth (OLTP) Layer

3. Source of Event Truth Layer (either CDC or subscribable events or both)

4. Metrics and Analytics (OLAP) Layer

For single datacenter I might choose Tarantool or Aerospike (too bad if you want to store more than your RAM you must use paid version) or Redis (master-slave) for cache layer, especially if the DNS is smart enough to route the client's request to nearest datacenter.

Tarantool can also be source of immediate truth, but the problem is it should be master-slave replication so it could be fault tolerant, but it seems there might have to do manual intervention to fallback to promote slave node as master and reroute other slave to recognize new master?

So the other choice is either TiDB (for consistent use case, lower write rate, more complex query use case) or ScyllaDB (for partition tolerant use case, higher write rate). Both are good in terms of availability. TiDB's TiFlash also good for analytics use case.

For source of event truth, we can use RedPanda for pubsub/MQ use cases, and Debezium (that requires Kafka/RedPanda) for change data capture from ScyllaDB, or TiCDC for TiDB to stream to RedPanda.

Lastly for analytics, we can use Clickhouse, also to store all structured logs that can be easily queried, or can also be Loki. For metrics might aggregate from RedPanda using MaterializeIO (too bad that cluster is paid).

So.. what are the combinations possible?

1. Tarantool (manual counter for metrics) + Clickhouse (manual publish logs and analytics), this one good only for single location/datacenter, unless the clients are all can hit proper server location (like game servers, or smart CDNs)

2. same as #1 but with RedPanda if have multiple service, all logs and events published thru RedPanda

3. Aerospike/Tarantool/Redis (manual counter for metrics) + TiDB + TiFlash (analytics) + Loki for logs

4. Aerospike/Tarantool/Redis (manual counter for metrics) + TIDB + TiCDC + ClickHouse (for logs and analytics)

5. Aerospike/Tarantool/Redis (cache only) + TiDB + TiFlash (analytics) + TiCDC + MaterializeIO (for metrics) + Loki (logs)

6. Aerospike/Tarantool/Redis (cache only) + TiDB + TiCDC + Clickhouse (analytics and logs) + MaterializeIO (for metrics)

7. Aerospike/Tarantool/Redis (cache only) + ScyllaDB + Debezium + RedPanda + Clickhouse (analytics and logs) + MaterializeIO (for metrics) 

for number #5-#7 you can remove the Aerospike/Tarantool part if no need to cache (consistency matters, and you have a very large cluster that can handle peaks).

wait, why don't you include full-text search use case? '__') ok, we can use Debezium that publish to ElasticSearch (might be overkill), or manual publish to TypeSense or MeiliSearch.

That's only for data part of the cluster, what about computation and presentation layer (backend/frontend)?

Backend of course I will use Golang or C#, frontend either Svelte (web only) or Flutter (mobile and web), Unity3D (game).

For object storage (if locally I would use MinIO) and CDN (CloudFlare for example), you can see my previous post.

So why you choose those?

1. Tarantool, one of the fastest in-mem, like 200K wps/qps but the cons I already mentioned above, the performance similar to Redis, but this one supports SQL.

2. Aerospike, also one of the fastest in-mem, like also 200K wps/qps last time I checked, also can do master-master replication, if I'm not mistaken, limited to 4 nodes for free verrsion, can set the replication factor, but see other cons I mentioned above.

3. TiDB, one newsql with automatic rebalance (one node died, and it would still works fast), can do around 30-50K single node last time I benchmarked, their benchmark mostly shows 30K-40K rps for writes, 120K rps for mixed read-write, 330K rps for read-only multi nodes benchmark, need more space? add more TiKV instance, need more compute? add more TiDB instance, need to do faster analytics queries? add TiFlash instance. But.. the bad part is, so many moving parts, you have to deploy TiDB (query engine), TiKV (storage), and PD (placement driver), also TiCDC and TiFlash if you need those too, also I'm not sure how it would perform at multi-DC use-cases. What's the managed alternative? AlloyDB-PostgreSQL (GCP) or TiDB-Cloud (AWS)

4. ScyllaDB, is faster version of Cassandra, most of benchmark can do 120K-400K insert and queries per second, one of the best database for mutli-DC, each keyspace can be set whether should replicate in which datacenter, how many the replication factor, consistency controlled in client-side, also can manage multiple view since 2 years ago (materialized view), so we don't have to create and maintain multiple table manually for each query pattern. What's the managed alternative? ScyllaDB-Cloud

5. RedPanda, is faster version of Kafka, last time I checked, one instance can receive 800K events per seconds, and publishes 4 millions events per second.

6. Clickhouse, is one of the best analytics database, can do a 1M batched ingestion per second in single node, can do complex queries really fast under a second (but depends on how big your data, but kind of query that if you did it in normal RDBMS would took minutes), the cons is one node can only handle 100 queries concurrently.

7. MaterializeIO, is like ksqldb but written in Rust, haven't checked the performance but they claimed they perform as fast as Redis.

What's other alternative? Yugabyte looks ok, especially the YCQL part that works like Cassandra/ScyllaDB. Especially yugabyte seems to be combining Redis, Postgres, and Cassandra in one deployment, but I like TiDB more because last time I checked, I need to config something to make it writable when one node died.

Proxy and load balancer? Caddy, Traefik, FabioLB (with Consul), or NATS (eg. 3 load balancer/custom api gateway that deployed in front, then it would serialize request to NATS inside DMZ, the worker/handler will receive that and return response that deserialized back to the loadbalancer/api gateway, that way load balancer doesn't need to know how exactly how many worker/handler, services can also communicate synchronously thru NATS without knowing each other service's IP address, and the worker/handler part can be scaled independently)

Deployment? Nomad, just rsync it to Jelastic, DockerCompose for dependencies (don't forget to bind the volume or your data will gone), Kubernetes

Tracker? maybe Pirsch

Why there's no __INSERT_DB_TYPE_HERE__? Graph database? because I rarely do recursive/graph queries, and especially I don't know which one that are best? Neo4J? DGraph? NebulaGraph? TigerGraph? AnzoGraph? TerminusDB? Age? JanusGraph? HugeGraph?

Have some other cool/high-performance tech stack suggestion? Chat with me at http://t.me/kokizzu (give proper intro tho, or I would think you are a crypto scam spammer XD)

2021-11-14

Databases with Automatic Rebalance Benchmark (TIDB vs YugabyteDB vs CockroachDB)

Automatic rebalance/repair/self-healing (we can remove or add new node, and it will distribute the data and rebalance itself, data are replicated to more than 1 node). Previous benchmark doesn't really care about this awesome feature (no more cutoff downtime to kill master instance and promote slave as master then switch every client to connect to new master -- if not using any proxy).

Some databases that I found that support this feature:

  1. Aerospike (in-mem kv database, community edition max 4 billion object) https://aerospike.com/products/product-matrix/
  2. Couchbase (document database, there's community edition) https://www.couchbase.com/products/editions
  3. MongoDB (document database, without SQL syntax)
  4. Cassandra (columnar database, CQL a subset of SQL)
  5. ScyllaDB (columnar database, cassandra-compatible, oss version max 5 node) https://www.scylladb.com/pricing/
  6. CockroachDB (postgresql compatible, there's core edition) https://www.cockroachlabs.com/compare/
  7. TiDB (mysql compatible)
  8. YugaByteDB (postgresql/cassandra/redis compatible)
  9. RavenDB (community edition max ram 6GB) https://ravendb.net/buy
  10. SingleStore/MemSQL (mysql compatible, free edition max 4 node) https://www.singlestore.com/free-software/
Reproducibility

The repository are here: https://github.com/kokizzu/hugedbbench on the 2021 folder. We're going to test local single (if possible) and multi server deployment using docker. Why using docker? because i don't want to ruin my computer/server with trash files they are creating in system directory (if any). Some of databases not included if not supporting SQL or if a license key required to start. Why only benchmarking 2 column? because it fit my project's most common use case, where there's 1 PK (bigint or string), and 1 unique key (mostly string), and the rest mostly some indexed or non-indexed column. Why are you even doing this? Just want to select the best thing for my next side project's techstack (and because my past companies I've work with seems love to move around database server location a lot).
The specs for the server that used in this benchmark: 32-core 128GB RAM 500GB NVMe disk.

CockroachDB

CockroachDB is one of NewSQL movement that support PostgreSQL syntax, to deploy in single node we can use docker compose. The UI for cluster monitor on port 8080 is quite ok :3 better than nothing.

Here's the result for 100 inserts x 1000 goroutines:

Version used: v21.1.11
CockroachDB InsertOne 10.034616078s
CockroachDB Count 42.326487ms
CockroachDB UpdateOne 12.804722812s
CockroachDB Count 78.221432ms
CockroachDB SelectOne 2.281355728s
CockroachDB Total 25.2420225s
$ sudo du -h --max-depth 0 2021/cockroachdb/cockroach1
442M    2021/cockroachdb/cockroach1

CockroachDB InsertOne 7.125466063s
CockroachDB Count 39.753102ms
CockroachDB UpdateOne 10.221870484s
CockroachDB Count 70.624908ms
CockroachDB SelectOne 2.196985441s
CockroachDB Total 19.655920219s
432M    2021/cockroachdb/cockroach1

# multiple cockroach docker but connect only into one
# seems high availability (>1 RF) turned on by default
# but you have to init the cluster manually after docker-compose up
CockroachDB InsertOne 13.979257573s
CockroachDB Count 46.824883ms
CockroachDB UpdateOne 1m22.941738013s
CockroachDB Count 42.374814ms
CockroachDB SelectOne 2.676679427s
CockroachDB Total 1m39.687566436s
433M    2021/cockroachdb/cockroach1
292M    2021/cockroachdb/cockroach2
222M    2021/cockroachdb/cockroach3

TiDB

TiDB is one of NewSQL movement that support MySQL syntax, the recommended way is using tiup command, but we're going to use docker so it would be fair with other database product. The official docker use 3 placement driver and 3 kv server, so I try that first. The cluster monitor in port 10080 but it blocked by chrome, so I moved it on 10081, it's very plaintexty compared to other products.

Version used: 5.7.25-TiDB-v5.0.1
TiDB InsertOne 14.063953386s
TiDB Count 32.523526ms
TiDB UpdateOne 11.329688001s
TiDB Count 49.320725ms
TiDB SelectOne 2.110410282s
TiDB Total 27.601866351s
$ sudo du -h --max-depth 0 2021/tidb/t*/
24G     2021/tidb/tikv0/
24G     2021/tidb/tikv1/
24G     2021/tidb/tikv2/
123M    2021/tidb/tipd0/
123M    2021/tidb/tipd1/
123M    2021/tidb/tipd2/

TiDB InsertOne 13.434256392s
TiDB Count 44.192782ms
TiDB UpdateOne 12.575839233s
TiDB Count 63.126285ms
TiDB SelectOne 2.00257672s
TiDB Total 28.134319527s
24G     2021/tidb/tikv0/
24G     2021/tidb/tikv1/
24G     2021/tidb/tikv2/
123M    2021/tidb/tipd0/
62M     2021/tidb/tipd1/
62M     2021/tidb/tipd2/

# reducing to single server mode (1 pd, 1 kv, 1 db), first run:
TiDB InsertOne 3.216365486s
TiDB Count 34.30629ms
TiDB UpdateOne 3.913131711s
TiDB Count 62.202395ms
TiDB SelectOne 1.991229179s
TiDB Total 9.233077269s
24G     2021/tidb/tikv0/
62M     2021/tidb/tipd0/

YugaByteDB

YugaByteDB is one of NewSQL movement that support PostgreSQL syntax, to deploy in single node we can use docker compose too. The cluster monitor on port :7000 is quite ok. The tmp directory mounted because if it isn't it would stuck starting on 2nd time unless the temporary file manually deleted. limits.conf applied.

Version used: 2.9.1.0
YugaByteDB InsertOne 11.402609701s
YugaByteDB Count 159.357304ms
YugaByteDB UpdateOne 19.232827282s
YugaByteDB Count 214.389496ms
YugaByteDB SelectOne 2.778803557s
YugaByteDB Total 33.834838111s
$ sudo du -h --max-depth 0 2021/yugabytedb/yb*1
25M     2021/yugabytedb/ybmaster1
519M    2021/yugabytedb/ybtserver1

YugaByteDB InsertOne 13.536083917s
YugaByteDB Count 202.381009ms
YugaByteDB UpdateOne 20.78337085s
YugaByteDB Count 190.119437ms
YugaByteDB SelectOne 2.849347721s
YugaByteDB Total 37.607747856s
25M     2021/yugabytedb/ybmaster1
519M    2021/yugabytedb/ybtserver1

# multiple ybtserver but only connect to one
# replication factor 1, first run:
YugaByteDB InsertOne 15.260747636s
YugaByteDB Count 66.599257ms
YugaByteDB UpdateOne 26.246382158s
YugaByteDB Count 63.119089ms
YugaByteDB SelectOne 3.213271599s
YugaByteDB Total 44.90095282s
25M     2021/yugabytedb/ybmaster1
242M    2021/yugabytedb/ybtserver1
156M    2021/yugabytedb/ybtserver2
132M    2021/yugabytedb/ybtserver3

# after changing replication factor to 2, first run:
YugaByteDB InsertOne 38.614091068s
YugaByteDB Count 76.615212ms
YugaByteDB UpdateOne 56.796680169s
YugaByteDB Count 84.35411ms
YugaByteDB SelectOne 3.14747611s
YugaByteDB Total 1m38.756226195s
26M     2021/yugabytedb/ybmaster1
343M    2021/yugabytedb/ybtserver1
349M    2021/yugabytedb/ybtserver2
349M    2021/yugabytedb/ybtserver3

# after changing replication factor to 3, first run:
YugaByteDB InsertOne 45.289805293s
YugaByteDB Count 97.112383ms
YugaByteDB UpdateOne 54.665380464s
YugaByteDB Count 64.206741ms
YugaByteDB SelectOne 3.125693618s
YugaByteDB Total 1m43.290014042s
26M     2021/yugabytedb/ybmaster1/
513M    2021/yugabytedb/ybtserver1/
512M    2021/yugabytedb/ybtserver2/
512M    2021/yugabytedb/ybtserver3/

Conclusion


Here's the recap of 100 records x 1000 goroutine insert/update/select duration, only for single instance:

Only Best of 2 runsCockroachDB
(single)
TiDB (single)YugaByteDB
(single)
InsertOne (s)7.1(best) 2.9(worst) 11.4
UpdateOne (s)10.2(best) 3.9(worst) 19.2
SelectOne (s)2.1(best) 1.9(worst) 2.7
Total (s)19.6(best) 9.2(worst) 33.8
Disk Usage (MB)(best) 432(worst) 24062544

So, at best, it roughly on average take 29 μs to insert, 39 μs to update, 19 μs to select one record.
Comparing only multi (RF=2+):

Only Best of 2 runsCockroachDB
(multi)
TiDB (multi)YugaByteDB
(multi)
YugaByteDB
(multi 2)
YugaByteDB
(multi 3)
InsertOne (s)12.5(best) 3.114.334.9(worst) 45.2
UpdateOne (s)(worst) 60.1(best) 4.119.652.754.6
SelectOne (s)(worst) 3.1(best) 2.12.8(worst) 3.12.9
Total (s)77.2(best) 9.540.891.7(worst) 103.2
Disk Usage (MB)1015(worst) 72247(best) 52110461563

So, at best, it roughly on average take 31 μs to insert, 41 μs to update, 21 μs to select one record.
Comparing only multi with replication factor with true HA:

Only Best of 2 runsCockroachDB
(multi)
TiDB (multi)YugaByteDB
(multi 3)
InsertOne (s)13.9(best) 3.1(worst) 45.2
UpdateOne (s)(worst) 82.9(best) 4.154.6
SelectOne (s)2.6(best) 2.12.9
Total (s)(worst) 99.6(best) 9.5(worst) 103.2
Disk Usage (MB)(best) 1015(worst) 722471563

It seems TiDB has most balanced performance in expense the need to have pre-allocated disk space, while CockroachDB has worst performance on multi-instance update task, and YugabyteDB has worst performance on multi-instance insert task.

What happened if we do the benchmark once more, remove one storage node (docker stop), then redo the benchmark (only for RF=2+)?

Yugabytedb test doesn't even entering the insert stage after 5 minutes '__') may be because of truncate is slow? so I changed the benchmark scenario only for yugabyte to be 1 node be killed after 2 seconds of insertion phase, but still yugabyte giving an error "ERROR: Timed out: Write RPC (request call id 3873) to 172.21.0.5:9100 timed out after 60.000s (SQLSTATE XX000)", it cannot complete. EDIT yugabyte staff on slack suggested that it should be using RF=3 so it would still survive when one node died.

Only 1 runCockroachDB
(multi, kill 1)
TiDB (multi, kill 1 tikv)TiDB (multi, kill 1 tipd)TiDB (multi, kill 1 tikv 1 tipd)YugaByteDB
(multi 3, kill 1)
InsertOne (s)(worst) 35.919.214.3(best) 9.234.8
UpdateOne (s)18.611.616.0(best) 9.9(worst) 68.8
SelectOne (s)4.01.9(best) 1.82.33.1
Total (s)58.832.934.0(best) 21.5(worst) 106.9
Disk Usage (MB)(best) 1076(worst) 7236472308723082659

TiDB seems to be the winner also for case when a node died, in expense of the need of 7 initial node (1 tidb [should be at least 2 for HA], 3 tipd, 3 tikv, but probably can be squeezed to be 1 tidb, 1 tipd, 2 tikv, since apparently the default replication factor is 3), where cockroachdb only need 3, and yugabytedb need 4 (1 ybmaster, 3 ybserver). Not sure tho what would happened if 1 tidb/ybmaster instance is died. The recap spreadsheet are here.

Next time we're gonna test how simple is it to add and remove node (and securely, if possible only limited set of servers can join without have to set firewall/DMZ to restrict unprivileged servers) then re-benchmark with more complex common use case (like UPSERT, range queries, WHERE-IN, JOIN, and secondary index). If automatic rebalance not in the requirement, I would still use Tarantool (since 2020.09) and Clickhouse (since 2021.04), but now I found one more new favorite automatic-rebalance database other than Aerospike (since 2016.11), :3 myahaha! So this is probably the reason lots of companies moving to TiDB.

Btw do not comment on this blog (since it's too much spammy comment and there's no notification whether new comment added), just use github issue or reddit instead.

UPDATE #1: redo the benchmark for all database after updating the limits.conf, TiDB improved by a lot, while CockroachDB remains the same except for update benchmark.
 
UPDATE #2: TiKV can be set to use only a little of initial disk space by setting "reserve-space" it would only use 4.9GB per TiKV after the test