Showing posts with label go. Show all posts
Showing posts with label go. Show all posts

2021-11-17

Alternative Strategy for Dependency Injection (lambda-returning vs function-pointer)

There's some common strategy for injecting dependency (one or sets of function) using interface, something like this:

type Foo interface{ Bla() string }
type RealAyaya struct {}
func(a *RealAyaya) Bla() {}
type MockAyaya struct {} // generated from gomock or others
func(a *MockAyaya) Bla() {}
// real usage:
deps := RealAyaya{}
deps.Bla()
// test usage:
deps := MockAyaya{}
deps.Bla()

and there's another one (dependency parameter on function returning a lambda):

type Bla func() string
type DepsIface interface { ... }
func NewBla(deps DepsIface) Bla {
  return func() string {
    // do something with deps
  }
// real usage:
bla := NewBla(realDeps)
res := bla()
// test usage:
bla := NewBLa(mockedOrFakeDeps)
res := bla()

and there other way by combining both fake and real implementation like this, or alternatively using proxy/cache+codegen if it's for 3rd party dependency.
and there other way (plugggable per-function level):

type Bla func() string
type BlaCaller struct {
  BlaFunc Bla
}
// real usage:
bla := BlaCaller{ BlaFunc: deps.SomeMethod }
res := bla.BlaFunc()
// test usage:
bla := BlaCaller{ BlaFunc: func() string { return `fake` } }
res := bla.BlaFunc()

Analysis


The first one is the most popular way, the 2nd one is one that I saw recently (that also being used in openApi/swagger codegen, i forgot which library), the bad part is that we have to sanitize the trace manually because it would show something like NewBla.func1 in the traces, and we have to use generated mock or implement everything if we have to test. Last style is what I thought when writing some task, where the specs still unclear whether I should:
1. query from local database
2. hit another service
3. or just a fake data (in the tests)
I can easily switch out any function without have to depend on whole struct or interface, and it would be still easy to debug (set breakpoint) and jump around the method, compared to generated mock or interface version.
Probably the bad part is, we have to inject every function one by one for each function that we want to call (which nearly equal effort as the 2nd one). But if that's the case, when your function requires 10+ other function to inject, maybe it's time to refactor?

The real use case would be something like this:

type LoanExpirationFunc func(userId string) time.Time 
type InProcess1 struct {
  UserId string 
// add more input here
  LoanExpirationFunc LoanExpirationFunc
  // add more injectable-function, eg. 3rd party hit or db read/save
}
type OutProcess1 struct {}
func Process1(in *InProcess1) (out *OutProcess1) {
  if ... // eg. validation
  x := in.LoanExpirationFunc(in.UserId) 
  // ... // do something
}

func defaultLoanExpirationFunc(userId string) time.Time {
  // 
eg. query from database
}

type thirdParty struct {} // to put dependencies
func NewThirdParty() (*thirdParty) { return &thirdParty{} }
func (t *thirdParty) extLoanExpirationFunc(userId string) time.Time {
  // eg. hit another service
}

// init input:
func main() {
  http.HandleFunc("/case1", func(w, r ...) {
    in := InProcess1{LoanExpirationFunc: defaultLoanExpirationFunc}
    in.ParseFromRequest(r)
    out := Process1(in)  
    out.WriteToResponse(w)
  })
  tp := NewThirdParty()
  http.HandleFunc("/case2", func(w, r ...) {
    in := InProcess1{LoanExpirationFunc: tp.extLoanExpirationFunc}
    in.ParseFromRequest(r)
    out := Process1(in)  
    out.WriteToResponse(w)
  })
}

// on test:
func TestProcess1(t *testing.T) {
  t.Run(`test one year from now`, func(t *testing.T) {
    in := inProcess1{LoanExpirationFunc: func(string) { return time.Now().Add(1, 0, 0) }}
    out := Process1(in)
    assert.Equal(t, out, ...)
  })
}

Haven't using this strategy extensively on new a project (since I just thought about this today and yesterday when creating horrid integration test), but I'll update this post when I found annoyance with this strategy.
 
UPDATE 2022: after using this strategy extensively for a while, this one is better than interface (especially when using IntelliJ), my tip: it would be better if you use function pointer name and injected function name with same name.

2021-09-11

Against Golang Interface{Method}-abuse/pollution

As you already know, after doing a lot of maintenance work of other people's code, I don't like to follow blindly so called "best practice" or popular practice that are proven painful in the long run when it's followed blindly/doesn't fit project's use case, eg.
  • using dynamically-typed language (JS, Python, PHP, Ruby, etc) just because it's the most popular language -- only for short/discardable project
  • mocking -- there's better way
  • microservice without properly splitting domain -- modular monolith is better for small teams, introducing network layer just to split a problem without properly assessing surely will be a hassle in a short and long run
  • overengineering -- eg. adding stack that you don't need when current stack suffice, for example,  dockerizing or kubernetesizing just because everyone using it, adding ElasticSearch just because it's search use case, but the records needs to be searched are very little and rps are very low, a more lightweight aproach more  make sense: eg. TypeSense or MeiliSearch or even database's built-in FTS are more make sense for lower rps target/simpler search feature.
  • premature "clean architecture" -- aka. over-layering everything that you'll are almost never replace -- dependency tracking is better
  • unevaluated standard -- sticking with standard just because it's a standard, just like being brainwashed/peer-pressured by dead people's will (tradition) without rethinking is it still make sense to be followed for this use case?
  • not making SRS/Software Requirement Specification (roles/who can do what action/API) and SDS/Software Design Specification (this action/API will mutate/command or read/query which datastore or hit which 3rd party) -- this helps new guy to be onboarded to the project really fast
I have one more unpopular opinion, interface (-overuse) in Golang is almost always bad for jumping around (jump to declaration-implementation) inside source code which causes everyday overhead when reading code and debugging. For example when you want to create a fake/mock/stub of certain method:

type Bla interface { 
  Get(string) string
  Set(string)
}

struct RealBla struct {} // wraps a 3rd party/client library
func (*RealBla) Get(string) string { return `` }
func (*RealBla) Set(string) { }

struct FakeBla struct {} // our fake/stub/mock implementation
func (*FakeBla) Get(string) string { return `` }
func (*
FakeBla) Set(string) { }

// usage

func TestBla(t *testing.T) {

   var b Bla = FakeBla{
...}
   // usually as data member of other method that depends on RealBla
   b.Set(...)
   x := b.Get(...)
   
}

func main() {
   var b Bla = RealBla{...}
   b.Set(...)
   x := b.Get(...)
}

the problem with this approach is, it's harder to jump around between declaration and implementation (usually RealBla that we want, not FakeBla), how often we switch implementation anyway? YAGNI (vs overengineering). It's better for our cognitive/understanding that we keep both coupled, this violates single responsibility principle from SOLID, but it's easier to reason/understand, since the real and fake are in the same file and near each other, so we can catch bug easily without have to switch, something like this:

struct BlaWrapper {
  // declare/use 3rd party client here
  UseFake bool
  // create fake/in-mem here
}

func (b *BlaWrapper) Get(s string) string {
  if b.
UseFake {
    // do with fake
    return
  }

  // do with real 3rd party
}

func (b  *BlaWrapper) Set(s string) {
  if b.UseFake {
    // do with fake
    return
  }

  // do with real 3rd party
}

// usage

func TestBla(t *testing.T) {
  var b = BlaWrapper{UseFake:true,
...}
  b.Set(...)
  x := b.Get(...)
}

func TestBla(t *testing.T) {
  var b = BlaWrapper{
...}
  b.Set(...)
  x := b.Get(...)
}


by doing this, we could compare easily between our fake and real implementation (you could easily spot the bug, whether your fake implementation differ way too much from real implementation), and we can still jump around simply by ctrl+click the IDE on that function since there's only 1 implementation. The only pros I could see from doing interface-based is when you are creating a 3rd party library (eg. io.Writer, io.Reader, etc) and you have more than 2 implementation (DRY only good when its more than 2), but since you're only making this for internal project that could be easily refactored within the project itself, it doesn't make sense to abuse interface. See more tips from this video: Go Worst Practice
After all being said, I won't use this kind of thing (UseFake property) for testing databases (2nd party), because I prefer to do integration (contract-based) testing instead of unit testing, since i'm using a fast database anyway (not a slow but popular RDBMSes).

2021-07-08

Prime Benchmark

So, yesterday I didn't touch my PC to do a prime number benchmark. And Here's the result for single threaded (only showing fastest implementation of each language) and multithreaded:

IndexImplementationSLabelPassesDurAlgoFaithfulBitPasses/Second
1cpp3flo80_2206675.00baseno144133.26760
10c2daniel202505.00wheelyes14049.85259
11zig3ManDeJ179645.00baseno13592.49823
12c2daniel176815.00wheelyes13536.07129
16rust1mike-b158045.01baseyes83152.68929
20assembly1rberge144345.00baseno82886.80000
22haskell1fatho/119595.00baseno82391.77321
32fortran1johand99875.00baseno11997.40000
36crystal1marghi86805.00baseyes11735.86981
38fsharp3dmanno77545.00baseyes1550.68897
40java1Mansen1488710.0baseyes1488.70000
41csharp1kinema72715.00baseyes1454.08077
43julia2epithe69535.00baseyes11390.55577
46go2ssoves61615.00baseyes11232.01471
51nodejs1rogier57485.00baseyes11149.43213
57lisp2mayerr51225.00baseno11024.19803
58typescript1marghi50315.00baseyes1006.20000
59d2Bradle50035.00baseyes11000.52396
61v1marghi43295.00baseyes865.80000
63lua2ben1je31595.00baseno1631.80000
64nim2beef3328715.00baseyes1574.02096
67cython1rpkak26595.00baseyes8531.64832
71basic1rberge24165.00wheelyes1483.00680
73assemblyscript1donmah423110.0baseyes423.05768
74python2ssoves19915.00baseyes8398.09742
80scala1rom1de12035.00baseyes240.55189
81pascal1rberge11625.00baseyes232.40000
82cobol1fvbake11575.00baseno8231.40000
83pony1marghi11445.00baseyes1228.80000
84swift1j-f1204610.0baseyes204.55332
85dart1eagere8245.00baseyes164.77795
86haxe1TayIor139210.0baseyes139.19035
88ada1BoopBe6615.00baseno132.02220
92octave1octave3135.00baseno62.54234
93postscript1epithe2165.01baseno843.08797
94ruby1rberge1195.01baseyes23.71935
95wren1marghi1115.00baseyes22.16446
96php1Dennis14310.0baseyes14.24667
97smalltalk1fvbake495.07baseyes19.66469
99mixal1rberge304.91baseno16.10998
100perl1marghi285.16baseyes5.42031
103r1fvbake75.43baseyes321.28842
104sql2fvbake65.43otherno321.10375
105tcl1fvbake65.47baseyes11.09589
111latex1tjol217.8baseno320.11224
112bash1bash110.6baseno0.09357

IndexImplementationSLabelPassesDurThreadAlgoFaithfulBitPasses/Second
1zig3ManDe1409105.04wheelno17045.26046
2cpp3flo802361845.08baseno15904.57992
3zig3ManDe1063995.04wheelno15319.64146
4zig3ManDe1010265.04wheelno15051.08785
5zig3ManDe840025.04wheelno14200.08320
6zig3ManDe1478225.08wheelno13695.38740
7zig3ManDe720215.04wheelno13600.73314
8zig3ManDe705225.04wheelno13525.83204
9zig3ManDe1341245.08wheelno13352.91894
10zig3ManDe598515.04baseno82992.45424
11c2danie1013915.08wheelyes12534.43285
12zig3ManDe982465.08wheelno12455.98299
13zig3ManDe981905.08wheelno12454.57327
14zig3ManDe481645.04baseno12408.10849
15zig3ManDe917455.08wheelno12293.50574
16zig3ManDe905985.08wheelno12264.84129
17c2danie881035.08wheelyes12199.97727
18zig3ManDe423185.04baseno12115.85768
19c2danie788585.08wheelyes11969.05838
20zig3ManDe684925.08baseno81712.11852
21c2danie637525.08wheelyes11591.86334
22rust1mike-590015.08baseyes81474.83765
23rust1mike-529795.08baseyes11324.32205
24c2danie498225.08baseyes11244.43126
25zig3ManDe497125.08baseno11242.59124
26c2danie246645.04wheelyes11233.15067
27rust1mike-492385.08baseyes11230.78145
28zig3ManDe456365.08baseno11140.80189
29c2danie223785.04wheelyes11118.88478
30c2danie203855.04wheelyes11019.22065
31c2danie202575.04wheelyes11012.82346
32c2danie151325.04baseyes1756.55294
33cpp2davep200485.08baseyes1501.18496
34d2Bradl195535.08baseyes1488.74973
35zig3ManDe92405.04baseno8461.94272
36zig3ManDe118345.08baseno8295.81332
37csharp1kinem31935.08wheelyes179.81766
38csharp1kinem29615.08baseyes174.02041

Raw result on this gist

2020-12-22

String Associative Array and CombSort Benchmark 2020 Edition

5 Years later since last string associative benchmark and lesser string associative benchmark (measuring string concat operation and built-in associative array set and get), numeric comb sort benchmark and string comb sort benchmark (measuring basic array random access, string conversion, and array swap for number and string), this year's using newer processor: AMD Ryzen 3 3100 running on 64-bit Ubuntu 20.04. Now with 10x more data to hopefully make the benchmark runs 10x slower (at least 1 sec), best of 3 runs.

alias time='/usr/bin/time -f "\nCPU: %Us\tReal: %es\tRAM: %MKB"'

$ php -v
PHP 7.4.3 (cli) (built: Oct  6 2020 15:47:56) ( NTS )

$ time php assoc.php 
637912 641149 67002
3808703 14182513 2343937
CPU: 1.25s      Real: 1.34s     RAM: 190644KB

$ python3 -V
Python 3.8.5

$ time python3 dictionary.py
637912 641149 67002
3808703 14182513 2343937
CPU: 5.33s      Real: 5.47s     RAM: 314564KB

$ ruby3.0 -v
ruby 3.0.0p0 (2020-12-25 revision 95aff21468) [x86_64-linux-gnu]

$ time ruby3.0 --jit hash.rb 
637912 641149 67002
3808703 14182513 2343937
CPU: 6.50s      Real: 5.94s     RAM: 371832KB

$ go version
go version go1.14.7 linux/amd64

$ time go run map.go
637912 641149 67002
3808703 14182513 2343937
CPU: 1.79s      Real: 1.56s     RAM: 257440KB

$ node -v       
v14.15.2

$ time node object.js
637912 641149 67002
3808703 14182513 2343937
CPU: 2.24s      Real: 2.21s     RAM: 326636KB

$ luajit -v 
LuaJIT 2.1.0-beta3 -- Copyright (C) 2005-2017 Mike Pall. http://luajit.org/

$ time luajit table.lua
637912  641149  67002
3808703 14182513        2343937
CPU: 4.11s      Real: 4.22s     RAM: 250828KB

$ dart --version
Dart SDK version: 2.10.4 (stable) (Unknown timestamp) on "linux_x64"

$ time dart map.dart
637912 641149 67002
3808703 14182513 2343937
CPU: 2.99s      Real: 2.91s     RAM: 385496KB

$ v version
V 0.2 36dcace

$ time v run map.v
637912, 641149, 67002
3808703, 14182513, 2343937
CPU: 4.79s      Real: 5.28s     RAM: 1470668KB

$ tcc -v
tcc version 0.9.27 (x86_64 Linux)

$ time tcc -run uthash.c
637912 641149 67002
3808703 14182513 2343937
Command exited with non-zero status 25

CPU: 2.52s      Real: 2.61s     RAM: 291912KB

export GOPHERJS_GOROOT="$(go1.12.16 env GOROOT)"
$ npm install --global source-map-support

$ goperjs version
GopherJS 1.12-3

$ time gopherjs 
637912 641149 67002
3808703 14182513 2343937

CPU: 14.13s     Real: 12.01s    RAM: 597712KB

$ java -version
java version "14.0.2" 2020-07-14
Java(TM) SE Runtime Environment (build 14.0.2+12-46)
Java HotSpot(TM) 64-Bit Server VM (build 14.0.2+12-46, mixed mode, sharing)

$ time java hashmap.java
637912 641149 67002
3808703 14182513 2343937

CPU: 5.18s      Real: 1.63s     RAM: 545412KB

The result shows a huge improvement for PHP since the old 5.4. NodeJS also huge improvement compared to old 0.10. The rest is quite bit the same. Also please keep note that Golang and V includes build/compile time not just run duration, and it seems V performance really bad when it comes to string operations (the compile itself really fast, less than 1s for 36dcace -- using gcc 9.3.0). 
Next we're gonna benchmark comb sort implementation. But this time we use jit version of ruby 2.7, since it's far way faster (19s vs 26s and 58s vs 66s for string benchmark), for ruby 3.0 we always use jit version since it's faster than non-jit. In case for C (TCC) which doesn't have built-in associative array, I used uthash, because it's the most popular. TinyGo does not complete first benchmark after more than 1000s, sometimes segfault. XS Javascript engine failed to give correct result, engine262 also failed to finish within 1000s.

LanguageCommand FlagsVersionAssocRAMNum CombRAMStr CombRAMTotalRAM
Gogo run1.14.71.56257,4400.7382,8444.74245,4327.03585,716
Gogo run1.15.61.73256,6200.7882,8964.86245,4687.37584,984
Nimnim r -d:release --gc:arc1.4.21.56265,1720.7979,2845.77633,6768.12978,132
Nimnim r -d:release --gc:orc1.4.21.53265,1600.9479,3805.83633,6368.30978,176
Javascriptnode14.15.22.21327,0480.87111,9726.13351,5209.21790,540
Crystalcrystal run --release0.35.11.81283,6481.44146,7006.09440,7969.34871,144
Javascript~/.esvu/bin/v88.9.2011.77177,7480.89105,4166.71335,2369.37618,400
Ctcc -run0.9.272.61291,9121.4580,8326.40393,35210.46766,096
Javajava14.0.2 2020-07-141.63545,4121.50165,8647.69743,57210.821,454,848
Nimnim r -d:release1.4.21.91247,4560.9679,4768.381,211,11611.251,538,048
Dartdart2.10.42.91385,4961.61191,9167.31616,71611.831,194,128
Pythonpypy7.3.1+dfsg-22.19331,7762.83139,7408.04522,64813.06994,164
Javascript~/.esvu/bin/chakra1.11.24.02.73487,4001.27102,19211.27803,16815.271,392,760
Javascript~/.esvu/bin/jsc2711175.90593,6240.68111,9729.09596,08815.671,301,684
Vv -prod run0.2 32091dd gcc-10.24.781,469,9321.8679,37614.061,560,51620.703,109,824
Lualuajit2.1.0-beta34.11250,8283.76133,42412.91511,19620.78895,448
Javascript~/.esvu/bin/smJavaScript-C86.0a15.61378,0641.4096,48013.81393,37620.82867,920
Vv -prod run0.2 32091dd gcc-9.35.051,469,9362.1479,40814.621,560,48421.813,109,828
Javascript~/.esvu/bin/graaljsCE Native 20.3.07.78958,3804.45405,90014.31911,22026.542,275,500
Gogopherjs run1.12-3 (node 14.15.2)11.76594,8962.04119,60418.46397,39632.261,111,896
Nimnim r1.4.26.60247,4443.0579,33231.851,211,20841.501,537,984
PHPphp7.4.31.34190,64410.11328,45234.51641,66445.961,160,760
Rubytruffleruby21.1.0-dev-c1517c5514.542,456,1563.09453,15229.273,660,28446.906,569,592
Crystalcrystal run0.35.15.69284,32812.00153,82831.69441,74049.38879,896
Javascript~/.esvu/bin/quickjs2020-11-083.90252,48423.4880,77234.80471,62462.18804,880
Vv run0.2 36dcace gcc-9.35.281,470,6686.6080,23258.991,561,17670.873,112,076
Lualua5.3.35.98366,51627.26264,64846.05864,30079.291,495,464
Rubyruby2.7.0p06.31371,45619.29100,53658.82694,56084.421,166,552
Pythonpython33.8.55.47314,56433.96404,97647.79722,82087.221,442,360
Rubyjruby9.2.9.07.451,878,18434.111,976,84459.837,115,448101.3910,970,476
Rubyruby3.0.0p05.94371,83224.8792,84474.321,015,096105.131,479,772
Gotinygo run0.16.0999.99318,1483.68300,548252.34711,3401256.011,330,036

Golang still the winner (obviously, since it's compiled), then Nim (Compiled), next best JIT or interpreter is NodeJS, Crystal (Compiled, not JIT), v8, followed Java, by TCC (Compiled) Dart, PyPy, V (Compiled, not JIT), LuaJIT, PHP, Ruby, and Python3. The recap spreadsheet can be accessed here

FAQ:
1. Why you measure the compile duration too? because developer experience also important (feedback loop), at least for me.
2. Why not warming up the VM first? each implementation have it's own advantage and disadvantage.
3. Why there's no C++, VB.NET, C#, D, Object-Pascal? don't want to compile things (since there's no build and run command in one flag).  
4. Why there's no Kotlin, Scala, Rust, Pony, Swift, Groovy, Julia, Crystal, or Zig? Too lazy to add :3 you can contribute tho (create a pull request, then I'll run the benchmark again as preferabbly as there's precompiled binary/deb/apt/ppa repository for the compiler/interpreter).

Contributors
ilmanzo (Nim, Crystal, D)