Map to Struct and Struct to Map Golang Benchmark 2022 Edition

Sometimes we want to convert from map to struct or struct to map (dictionary in other language), or even struct to struct. There's some library that can help us doing this, for example structs, mapstructure, copier, or smapping. We could also utilize serialization and deserialization libraries to do this. With caveats, that some serialization format (eg. JSON) doesn't allow integer larger than 2^53 for example.

Here's benchmark that I run this morning:

map to struct	total	ns/op	B/op	allocs/op
M2S_GoccyGoJson_MarshalUnmarshal-32	6,661,932	517	80	3
M2S_JsonIteratorGo_MarshalUnmarshal-32	4,892,611	724	196	8
M2S_VmihailencoMspackV5_MarhsalUnmarshal-32	4,572,597	741	188	5
M2S_FxamackerCbor_MarshalUnmarshal-32	4,418,558	799	120	8
M2S_SurrealdbCork_EncodeDecode-32	3,080,282	1,080	1,217	6
M2S_GopkgInMgoV2Bson_MarshalUnmarshal-32	3,227,905	1,092	232	13
M2S_ShamatonMsgpackV2_MarshalUnmarshal-32	3,062,677	1,161	956	15
M2S_MitchellhMapstructure_Decode-32	2,487,428	1,395	720	18
M2S_MongoDriverBson_MarshalUnmarshal-32	2,477,983	1,459	414	14
M2S_KokizzuJson5b_MarshalUnmarshal-32	1,987,240	1,711	632	16
M2S_EncodingJson_MarshalUnmarshal-32	2,056,944	1,780	600	16
M2S_EtNikBinngo_MarshalUnmarshal-32	1,985,595	1,857	425	39
M2S_PquernaFfjson_MarshalUnmarshal-32	1,739,968	1,986	609	16
M2S_UngorjiGocodec_BincEncodeDecode-32	1,401,453	2,582	4,340	23
M2S_UngorjiGoCodec_CborEncodeDecode-32	1,304,828	2,636	4,340	23
M2S_PelletierGoTomlV2_MarshalUnmarshal-32	1,284,037	2,787	1,600	27
M2S_UngorjiGocodec_SimpleEncodeDecode-32	1,295,926	2,810	4,340	23
M2S_UngorjiGocodec_JsonEncodeDecode-32	1,000,000	3,028	4,956	25
M2S_IchibanTnetstrings_MarshalUnmarshal-32	749,947	5,056	9,329	48
M2S_BurntSushiToml_EncodeUnmarshal-32	425,335	8,065	7,958	71
M2S_HjsonHjsonGoV4_MarshalUnmarshal-32	355,784	10,870	3,936	78
M2S_GopkgInYamlV3_MarshalUnmarshal-32	271,190	13,524	14,112	80
M2S_DONUTSLz4Msgpack_MarshalUnmarshal-32	240,619	15,498	1,264	16
M2S_GoccyGoYaml_MarshalUnmarshal-32	214,776	16,192	7,821	214
M2S_GhodssYaml_MarshalUnmarshal-32	156,412	23,347	21,378	161
M2S_NaoinaToml_MarshalUnmarshal-32	57,607	58,331	398,544	77
struct to map	total	ns/op	B/op	allocs/op
S2M_MitchellhMapstructure_Decode-32	5,055,402	716	536	12
S2M_GoccyGoJson_MarshalUnmarshal-32	4,660,224	747	522	12
S2M_JsonIteratorGo_MarshalUnmarshal-32	4,283,262	835	505	14
S2M_VmihailencoMspackV5_MarhsalUnmarshal-32	4,009,863	908	607	12
S2M_FxamackerCbor_MarshalUnmarshal-32	3,562,352	1,023	452	11
S2M_ShamatonMsgpackV2_MarshalUnmarshal-32	3,180,010	1,089	556	15
S2M_GopkgInMgoV2Bson_MarshalUnmarshal-32	3,047,396	1,145	528	15
S2M_SurrealdbCork_EncodeDecode-32	2,976,328	1,196	1,611	12
S2M_EncodingJson_MarshalUnmarshal-32	1,914,165	1,782	688	18
S2M_PquernaFfjson_MarshalUnmarshal-32	1,911,950	1,845	697	18
S2M_EtNikBinngo_MarshalUnmarshal-32	1,948,802	1,859	768	45
S2M_KokizzuJson5b_MarshalUnmarshal-32	1,888,774	1,884	960	20
S2M_MongoDriverBson_MarshalUnmarshal-32	1,857,649	1,995	759	18
S2M_PelletierGoTomlV2_MarshalUnmarshal-32	1,244,012	2,864	1,800	31
S2M_UngorjiGocodec_BincEncodeDecode-32	1,000,000	3,234	4,888	34
S2M_UngorjiGoCodec_CborEncodeDecode-32	989,671	3,358	4,888	34
S2M_UngorjiGocodec_SimpleEncodeDecode-32	1,000,000	3,400	4,888	34
S2M_UngorjiGocodec_JsonEncodeDecode-32	912,512	3,639	5,504	36
S2M_IchibanTnetstrings_MarshalUnmarshal-32	776,796	4,744	9,561	46
S2M_BurntSushiToml_EncodeUnmarshal-32	447,216	8,538	8,231	73
S2M_HjsonHjsonGoV4_MarshalUnmarshal-32	389,476	9,416	3,868	66
S2M_GopkgInYamlV3_MarshalUnmarshal-32	315,939	13,338	14,400	81
S2M_DONUTSLz4Msgpack_MarshalUnmarshal-32	242,330	14,298	744	16
S2M_GoccyGoYaml_MarshalUnmarshal-32	230,042	14,919	7,580	202
S2M_GhodssYaml_MarshalUnmarshal-32	151,023	22,682	21,441	161
S2M_NaoinaToml_MarshalUnmarshal-32	60,916	52,047	398,112	80
struct to struct	total	ns/op	B/op	allocs/op
S2S_GoccyGoJson_MarshalUnmarshal-32	12,046,497	317	112	4
S2S_ShamatonMsgpackV2_MarshalUnmarshal-32	7,897,488	458	148	6
S2S_JsonIteratorGo_MarshalUnmarshal-32	7,853,592	494	92	6
S2S_FxamackerCbor_MarshalUnmarshal-32	7,038,808	511	80	5
S2S_GopkgInMgoV2Bson_MarshalUnmarshal-32	5,105,343	715	144	9
S2S_VmihailencoMspackV5_MarhsalUnmarshal-32	4,549,700	818	213	6
S2S_MongoDriverBson_MarshalUnmarshal-32	3,560,946	1,019	321	8
S2S_EncodingJson_MarshalUnmarshal-32	2,731,051	1,313	304	9
S2S_PquernaFfjson_MarshalUnmarshal-32	2,734,357	1,330	304	9
S2S_KokizzuJson5b_MarshalUnmarshal-32	2,594,728	1,343	504	9
S2S_SurrealdbCork_EncodeDecode-32	2,555,745	1,397	1,241	7
S2S_EtNikBinngo_MarshalUnmarshal-32	1,995,468	1,840	400	41
S2S_PelletierGoTomlV2_MarshalUnmarshal-32	1,460,683	2,459	1,440	23
S2S_UngorjiGocodec_SimpleEncodeDecode-32	1,205,648	2,919	4,364	24
S2S_UngorjiGoCodec_CborEncodeDecode-32	1,290,734	2,920	4,364	24
S2S_UngorjiGocodec_BincEncodeDecode-32	1,207,327	3,007	4,364	24
S2S_UngorjiGocodec_JsonEncodeDecode-32	1,000,000	3,223	4,980	26
S2S_IchibanTnetstrings_MarshalUnmarshal-32	722,493	4,950	9,289	47
S2S_BurntSushiToml_EncodeUnmarshal-32	398,366	8,458	7,918	72
S2S_HjsonHjsonGoV4_MarshalUnmarshal-32	304,369	11,189	4,578	79
S2S_DONUTSLz4Msgpack_MarshalUnmarshal-32	282,505	12,215	237	7
S2S_GopkgInYamlV3_MarshalUnmarshal-32	279,332	12,541	14,016	76
S2S_GoccyGoYaml_MarshalUnmarshal-32	211,952	15,542	7,982	208
S2S_GhodssYaml_MarshalUnmarshal-32	160,660	23,148	21,073	154
S2S_NaoinaToml_MarshalUnmarshal-32	64,468	59,672	399,065	83

The repository and the always updated result is here, feel free to add your own serialization/deserialization library. As we can see, goccy-gojson is the fastest among all, too bad if you store int64 larger than 2^53 it give wrong result. So it's better to use second best and all rounder vmihailenco-msgpack, or for specific use case struct to map/struct is mapstructure.

Here's the top ranking:

Ser/Deser	M2S	S2M	S2S
GoccyGoJson	1	2	2
JsonIteratorGo	2	3	4
MitchellhMapstructure	8	1	1
VmihailencoMspackV5	3	4	7
FxamackerCbor	4	5	5
ShamatonMsgpackV2	7	6	3
SurrealdbCork	5	8	12

Getting started with InfluxDB

Today we're gonna learn about InfluxDB, a time series database that already been a standard for open source metric storage, a part of TICK stack (telegraf -- a metric collector like datadog-agent/fluentd, influx -- the database, chronograf -- visualizer like grafana, kapacitor -- an alert manager like prometheus alert-manager also for ETL). To install influx use this steps from this link (there's also docker). What's the cons of influx compared to other solution? Datadog is (very very) expensive, Prometheus seems too kube-oriented, Influx open source doesn't support HA and clustering, that's why I would rather use Clickhouse for time series / log collection (as data sink) and aggregate those logs as metrics (materialized view) and copy periodically to faster database like Tarantool. Influx have 2 query syntax, one is SQL-like called InfluxQL and other one Javascript+FP-like called Flux (it has |> pipe like most FP). Here's the example of docker and the query language:

docker run -d -e INFLUXDB_ADMIN_USER:user1 -e INFLUXDB_ADMIN_PASSWORD:pass1 --name influxdb1 influxdb

docker exec -it influxdb1 influx -host 127.0.0.1 -port 8086 -username user1 -password pass1

show databases
create database db1
use db1

-- show all tables (=measurements)
show measurements

-- tag/index always string, field can be float, int, bool, etc
insert into table1,tag1=a,tag2=b field1=1,field2=2

-- if not set, default time is in nanosecond of insert time
select * from "table1"
select * from table1 where tag1='a'

-- describe columns from all table:
show tag keys
show field keys from "table1"

-- select distinct tag from table1
show tag values from "table1" with key in ("tag1")

-- automatically delete after N days, default 0 = never delete
-- shard by N hours, default 168 hours (a week)
show retention policies

-- like partition in other database
show shards
show shard groups

I guess that's it what you need to know to get started with InfluxDB. If you're looking one that comparable with Clickhouse, you can check TDEngine or TiFlash.

Techempower Framework Benchmark Round 21

 The result for Techempower framework benchmark round 21 is out, as usual the most important benchmark is the update and multi query benchmark:

The top rankers are Rust, JS (cheating), Java, C++, C#, PHP, C, Scala, Kotlin, and Go. For multiple queries:

Top rankers are Rust, Java, JS (cheating), Scala, Kotlin, C++, C#, and Go. These benchmark shows how efficient their database driver (which mostly the biggest bottleneck), and how much overhead the framework of each language (including serialization, alloc/GC, async-I/O, etc).

For memory usage and CPU utilization you can check here https://ajdust.github.io/tfbvis/?testrun=Citrine_started2022-06-30_edd8ab2e-018b-4041-92ce-03e5317d35ea&testtype=update&round=false