2021-07-30

Mock vs Fake and Classical Testing

Motivation of this article is to promote less painful way of testing, structuring codes, and less broken test when changing logic/implementation details (only changing the logic not changing the input/output). This post recapping past ~4 years compilation of articles that conclude that Fake > Mock, Classical Test > Mock Test from other developers that realize the similar pain points of popular approach (mock).

Mock Approach

Given a code like this:

type Obj struct {
*sql.DB // or Provider
}
func (o *Obj) DoMultipleQuery(in InputStruct) (out OutputStruct, err error) {
... = o.DoSomeQuery()
... = o.DoOtherQuery()
}

I’ve seen code to test with mock technique like this:

func TestObjDoMultipleQuery(t *testing.T) {
o := Obj{mockProvider{}}
testCases := []struct {
name string
mockFunc func(sqlmock.Sqlmock, *gomock.Controller) in InputStruct out OutputStruct
shouldErr bool
} {
{
name: `bast case`,
mockFunc: func(db sqlmock.SqlMock, c *gomock.Controller) {
db.ExpectExec(`UPDATE t1 SET bla = \?, foo = \?, yay = \? WHERE bar = \? LIMIT 1`).
WillReturnResult(sqlmock.NewResult(1,1))
db.ExpectQuery(`SELECT a, b, c, d, bar, bla, yay FROM t1 WHERE bar = \? AND state IN \(1,2\)`).
WithArgs(3).
WillReturnRows(sqlmock.NewRows([]string{"id", "channel_name", "display_name", "color", "description", "active", "updated_at"}).
AddRow("2", "bla2", "Bla2", "#0000", "bla bla", "1", "2021-05-18T15:04:05Z").
AddRow("3", "wkwk", "WkWk", "#0000", "wkwk", "1", "2021-05-18T15:04:05Z"))
...
}, in: InputStruct{...}, out: OutputStruct{...},
wantErr: false,
},
{
... other cases
},
} for _, tc := range testCases { t.Run(tc.name, func(t *testing.T){ ... // prepare mock object o := Obj{mockProvider} out := o.DoMultipleQueryBusinessLogic(tc.in) assert.Equal(t, out, tc.out) }) }
}

This approach has pros and cons:

+ could check whether has typos (eg. add one character in the original query, this test would detect the error)

+ could check whether some queries are properly called, or not called but expected to be called

+ unit test would always faster than integration test

- testing implementation detail (easily break when the logic changed)

- cannot check whether the SQL statements are correct

- possible coupled implementation between data provider and business logic

- duplicating work between original query and the regex-version of query, which if add a column, we must change both implementation

For the last cons, we can change it to something like this:

db.ExpectQuery(`SELECT.+FROM t1.+`).
WillReturnRows( ... )

This approach has pros and cons:

+ not deduplicating works (since it just a simplified regex of the full SQL statements

+ still can check whether queries properly called or not

+ unit test would always faster than integration test

- testing implementation detail (easily break when the logic changed)

- cannot detect typos/whether the query no longer match (eg. if we accidentally add one character on the original query that can cause sql error)

- cannot check correctness of the SQL statement

- possible coupled implementation between data provider and business logic

We could also create a helper function to replace the original query to regex version:

func SqlToRegexSql(sql string) string {
return // replace special characters in regex (, ), ?, . with escaped version
}
db.ExpectQuery(SqlToRegexSql(ORIGINAL_QUERY)) ...

This approach has same pros and cons as previous approach.

Fake Approach

Fake testing use classical approach, instead of checking implementation detail (expected calls to dependency), we use a compatible implementation as dependency (eg. a slice/map of struct for database table/DataProvider)

Given a code like this:

type Obj struct {
FooDataProvider // interface{UpdateFoo,GetFoo,...}
}
func (o *Obj) DoBusinessLogic(in *Input) (out *Output,err error) {
... = o.UpdateFoo(in.bla)
... = o.GetFoo(in.bla)
...
}

It’s better to make a fake data provider like this:

type FakeFooDataProvider struct {
Rows map[int]FooRow{} // or slice
}
func (f *FakeFooDataProvider) UpdateFoo(a string) (...) { /* update Rows */}
func (f *FakeFooDataProvider) GetFoo(a string) (...) { /* get one Rows */}
... // insert, delete, count, get batched/paged

So in the test, we can do something like this:

func TestObjDoBusinessLogic(t *testing.T) {
o := Obj{FakeFooDataProvider{}}
testCases := []struct{
name string
in Input
out Ouput
shouldErr bool
} {
{
name: `best case`,
in: Input{...},
out: Output{...},
shouldErr: false,
},
{
...
},
} for _, tc := range testCases { t.Run(tc.name, func(t *testing.T){ out := o.DoBusinessLogic(tc.in) assert.Equal(t, tc.out, out) }) }
}

This approach have pros and cons:

+ testing behavior (this input should give this output) instead of implementation detail (not easily break/no need to modify the test when algorithm/logic changed)

+ unit test would always faster than integration test

- cannot check whether the queries are called or not called but expected to be called

- double work in Golang (since there’s no generic/template yet, go 1.18 must wait Feb 2022), must create minimal fake implementation (map/slice) that simulate basic database table logic, or if data provider not separated between tables (repository/entity pattern) must create a join logic too – better approach in this case is to always create Insert, Update, Delete, GetOne, GetBatch instead of joining.

+ should be no coupling between queries and business logic

Cannot check whether queries in data provider are correct (which should not be the problem of this unit, it should be DataProvider integration/unit test’s problem, not this unit)

Classical Approach for DataProvider

It’s better to test the queries using classical (black box) approach integration test instead of mock (white box), since mock and fake testing can only test the correctness of business logic, not logic of the data provider that mostly depend to a 2nd party (database). Fake testing also considered a classical approach, since it test input/output not implementation detail.

Using dockertest when test on local and gitlab-ci service when test on pipeline, can be something like this:

var testDbConn *sql.DB
func TestMain(m *testing.M) int { // called before test
if env == `` || env == `development` { // spawn dockertest, return connection to dockertest
prepareDb(func(db *sql.DB){
testDbConn = db
if db == nil {
return 0
}
return m.Run()
})
} else {
// connect to gitlab-ci service var err error testDbConn, err = ... // log error
}
}
func TestDataProviderLogic(t *testing.T) {
if testDbConn == nil {
if env == `` || env == `development` || env == `test` {
t.Fail()
}
return
}
f := FooDataProvider{testDbConn}
f.InitTables()
f.MigrateTables() // if testing migration
// test f.UpdateFoo, f.GetFoo, ...
}

Where the prepareDb function can be something like this (taken from dockertest example):

func prepareDb(onReady func(db *sql.DB) int) {
const dockerRepo = `yandex/clickhouse-server`
const dockerVer = `latest`
const chPort = `9000/tcp`
const dbDriver = "clickhouse"
const dbConnStr = "tcp://127.0.0.1:%s?debug=true"
var err error
if globalPool == nil {
globalPool, err = dockertest.NewPool("")
if err != nil {
log.Printf("Could not connect to docker: %s\n", err)
return
}
}
resource, err := globalPool.Run(dockerRepo, dockerVer, []string{})
if err != nil {
log.Printf("Could not start resource: %s\n", err)
return
}
var db *sql.DB
if err := globalPool.Retry(func() error {
var err error
db, err = sql.Open(dbDriver, fmt.Sprintf(dbConnStr, resource.GetPort(chPort)))
if err != nil {
return err
}
return db.Ping()
}); err != nil {
log.Printf("Could not connect to docker: %s\n", err)
return
}
code := onReady(db)
if err := globalPool.Purge(resource); err != nil {
log.Fatalf("Could not purge resource: %s", err)
}
os.Exit(code)
}

In the pipeline the .gitlab-ci.yml file can be something like this for PostgreSQL (use tmpfs/inmem version for database data directory to make it faster):

test:
stage: test
image: golang:1.16.4
dependencies: []
services:
- postgres:13-alpine # TODO: create a tmpfs version
tags:
- cicd
variables:
ENV: test
POSTGRES_DB: postgres
POSTGRES_HOST: postgres
POSTGRES_PASSWORD: postgres
POSTGRES_PORT: "5432"
POSTGRES_USER: postgres
script:
- source env.sample
- go test

The dockerfile for tmpfs database if using MySQL can be something like this:

FROM circleci/mysql:5.5

RUN echo '\n\
[mysqld]\n\
datadir = /dev/inmemory/mysql\n\
' >> /etc/mysql/my.cnf

Or for MongoDB:

FROM circleci/mongo:3.6.9

RUN sed -i '/exec "$@"/i mkdir \/dev\/inmemory\/mongo' /usr/local/bin/docker-entrypoint.sh

CMD ["mongod", "--nojournal", "--noprealloc", "--smallfiles", "--dbpath=/dev/inmemory/mongo"]

The benefit of this classical integration test approach:

+ high confidence that your SQL statements are correct, can detect typos (wrong column, wrong table, etc)

+ isolated test, not testing business logic but only data provider layer, also can test for schema migrations

- not a good approach for database with eventual consistency (eg. Clickhouse)

- since this is an integration test,  it would be slower than mock/fake unit test (1-3s+ total delay overhead when spawning docker)

Conclusion

  1. use mock for databases with eventual consistency

  2. prefer fake over mock for business logic correctness because it’s better for maintainability to test behavior (this input should give this output), instead of implementation details

  3. prefer classical testing over mock testing for checking data provider logic correctness

References

(aka confirmation bias :3)

https://martinfowler.com/articles/mocksArentStubs.html
https://stackoverflow.com/questions/1595166/why-is-it-so-bad-to-mock-classes
https://medium.com/javascript-scene/mocking-is-a-code-smell-944a70c90a6a
https://chemaclass.medium.com/to-mock-or-not-to-mock-af995072b22e
https://accu.org/journals/overload/23/127/balaam_2108/
https://news.ycombinator.com/item?id=7809402
https://philippe.bourgau.net/careless-mocking-considered-harmful/
https://debugged.it/blog/mockito-is-bad-for-your-code/
https://engineering.talkdesk.com/double-trouble-why-we-decided-against-mocking-498c915bbe1c
https://blog.thecodewhisperer.com/permalink/you-dont-hate-mocks-you-hate-side-effects
https://agilewarrior.wordpress.com/2015/04/18/classical-vs-mockist-testing/
https://www.slideshare.net/davidvoelkel/mockist-vs-classicists-tdd-57218553
https://www.thoughtworks.com/insights/blog/mockists-are-dead-long-live-classicists
https://stackoverflow.com/questions/184666/should-i-practice-mockist-or-classical-tdd
https://bencane.com/2020/06/15/dont-mock-a-db-use-docker-compose/
https://swizec.com/blog/what-i-learned-from-software-engineering-at-google/#stubs-and-mocks-make-bad-tests

https://www.freecodecamp.org/news/end-to-end-api-testing-with-docker/
https://medium.com/@june.pravin/mocking-is-not-practical-use-fakes-e30cc6eaaf4e 
https://www.c-sharpcorner.com/article/stub-vs-fake-vs-spy-vs-mock/

2021-07-08

Prime Benchmark

So, yesterday I didn't touch my PC to do a prime number benchmark. And Here's the result for single threaded (only showing fastest implementation of each language) and multithreaded:

IndexImplementationSLabelPassesDurAlgoFaithfulBitPasses/Second
1cpp3flo80_2206675.00baseno144133.26760
10c2daniel202505.00wheelyes14049.85259
11zig3ManDeJ179645.00baseno13592.49823
12c2daniel176815.00wheelyes13536.07129
16rust1mike-b158045.01baseyes83152.68929
20assembly1rberge144345.00baseno82886.80000
22haskell1fatho/119595.00baseno82391.77321
32fortran1johand99875.00baseno11997.40000
36crystal1marghi86805.00baseyes11735.86981
38fsharp3dmanno77545.00baseyes1550.68897
40java1Mansen1488710.0baseyes1488.70000
41csharp1kinema72715.00baseyes1454.08077
43julia2epithe69535.00baseyes11390.55577
46go2ssoves61615.00baseyes11232.01471
51nodejs1rogier57485.00baseyes11149.43213
57lisp2mayerr51225.00baseno11024.19803
58typescript1marghi50315.00baseyes1006.20000
59d2Bradle50035.00baseyes11000.52396
61v1marghi43295.00baseyes865.80000
63lua2ben1je31595.00baseno1631.80000
64nim2beef3328715.00baseyes1574.02096
67cython1rpkak26595.00baseyes8531.64832
71basic1rberge24165.00wheelyes1483.00680
73assemblyscript1donmah423110.0baseyes423.05768
74python2ssoves19915.00baseyes8398.09742
80scala1rom1de12035.00baseyes240.55189
81pascal1rberge11625.00baseyes232.40000
82cobol1fvbake11575.00baseno8231.40000
83pony1marghi11445.00baseyes1228.80000
84swift1j-f1204610.0baseyes204.55332
85dart1eagere8245.00baseyes164.77795
86haxe1TayIor139210.0baseyes139.19035
88ada1BoopBe6615.00baseno132.02220
92octave1octave3135.00baseno62.54234
93postscript1epithe2165.01baseno843.08797
94ruby1rberge1195.01baseyes23.71935
95wren1marghi1115.00baseyes22.16446
96php1Dennis14310.0baseyes14.24667
97smalltalk1fvbake495.07baseyes19.66469
99mixal1rberge304.91baseno16.10998
100perl1marghi285.16baseyes5.42031
103r1fvbake75.43baseyes321.28842
104sql2fvbake65.43otherno321.10375
105tcl1fvbake65.47baseyes11.09589
111latex1tjol217.8baseno320.11224
112bash1bash110.6baseno0.09357

IndexImplementationSLabelPassesDurThreadAlgoFaithfulBitPasses/Second
1zig3ManDe1409105.04wheelno17045.26046
2cpp3flo802361845.08baseno15904.57992
3zig3ManDe1063995.04wheelno15319.64146
4zig3ManDe1010265.04wheelno15051.08785
5zig3ManDe840025.04wheelno14200.08320
6zig3ManDe1478225.08wheelno13695.38740
7zig3ManDe720215.04wheelno13600.73314
8zig3ManDe705225.04wheelno13525.83204
9zig3ManDe1341245.08wheelno13352.91894
10zig3ManDe598515.04baseno82992.45424
11c2danie1013915.08wheelyes12534.43285
12zig3ManDe982465.08wheelno12455.98299
13zig3ManDe981905.08wheelno12454.57327
14zig3ManDe481645.04baseno12408.10849
15zig3ManDe917455.08wheelno12293.50574
16zig3ManDe905985.08wheelno12264.84129
17c2danie881035.08wheelyes12199.97727
18zig3ManDe423185.04baseno12115.85768
19c2danie788585.08wheelyes11969.05838
20zig3ManDe684925.08baseno81712.11852
21c2danie637525.08wheelyes11591.86334
22rust1mike-590015.08baseyes81474.83765
23rust1mike-529795.08baseyes11324.32205
24c2danie498225.08baseyes11244.43126
25zig3ManDe497125.08baseno11242.59124
26c2danie246645.04wheelyes11233.15067
27rust1mike-492385.08baseyes11230.78145
28zig3ManDe456365.08baseno11140.80189
29c2danie223785.04wheelyes11118.88478
30c2danie203855.04wheelyes11019.22065
31c2danie202575.04wheelyes11012.82346
32c2danie151325.04baseyes1756.55294
33cpp2davep200485.08baseyes1501.18496
34d2Bradl195535.08baseyes1488.74973
35zig3ManDe92405.04baseno8461.94272
36zig3ManDe118345.08baseno8295.81332
37csharp1kinem31935.08wheelyes179.81766
38csharp1kinem29615.08baseyes174.02041

Raw result on this gist