2017-11-07

Elixir/Erlang better than Go, really?

The microbenchmark result for spawning short-lived concurrent process:

# Go 1.8.0
$ go build test.go ; for k in 5 50 500 5000 50000 500000; do echo -n $k; time ./test $k > /dev/null; done

 5
CPU: 0.00s      Real: 0.00s     RAM: 2080KB
50
CPU: 0.06s      Real: 0.01s     RAM: 3048KB
500
CPU: 0.61s      Real: 0.12s     RAM: 7760KB
5K
CPU: 6.02s      Real: 1.23s     RAM: 17712KB # 17 MB
50K
CPU: 62.30s     Real: 12.53s    RAM: 207720KB # 207 MB

 500K # this is 10x more! 
CPU: 649.47s    Real: 131.53s   RAM: 3008180KB # 3 GB

 # Elixir 1.4.2 (erts-8.2.2)
$ for k in 5 50 500 5000 50000 ; do echo -n $k; time elixir --erl "+P 90000000" test.exs $k > /dev/null; done

 5
CPU: 0.53s      Real: 0.50s     RAM: 842384KB # 842 MB
50
CPU: 1.50s      Real: 0.62s     RAM: 934276KB # 934 MB
500
CPU: 11.92s     Real: 2.53s     RAM: 1675872KB # 1.6 GB
5K
CPU: 122.65s    Real: 20.20s    RAM: 4336116KB # 4.3 GB
50K 
CPU: 1288.65s   Real: 209.66s   RAM: 6573560KB # 6.5 GB

You can find the code here. In terms of performance and memory usage, it's not really, but you can argue about anything else.
at No comments:

2017-07-31

Tricks to be Motivated

Recently I being so much unmotivated with my work, and I think I know the reason, but anyway, these are the generic things to get things done:

  1. Buy a pen, and small notebook, seriously! Make a TODO list, break into very small things to be done in a day
  2. Have enough sugar and sleep, lack of sleep/concentration/focus is bad, as bad as alcohol addiction
  3. Do your tasks first before doing unimportant tasks (facebook, tweeting, watching movies, etc), make those unimportant things as reward
  4. Do not imagine yourself reaching the success, probably better imagining if you are fail and be afraid of it
  5. Your willpower is unlimited, and can be strengthened by doing more things that requires willpower, as willpower is contagious
  6. Surround yourself with people with same goals, or at least watch motivational people that work hard to reach their goals/dreams, be envious!
  7. Forgive yourself, guilt makes you want instant gratification that produces more guilt
  8. Do less decision, always go for option A, be courageous, set simple rules
  9. Walk/start first, motivation will comes later
I think that's it, these article summarized from videos about motivation and willpower I found on youtube.





at No comments:

2017-06-14

Cross Platform Game Development

So I was looking for cross-platform game development, found some that are good for Indie game dev development, such as V-Play that uses QML (I like Qt), but the review for the games it produced is quite bad (crashes a lot), so I look another one and found Cocos2D-JS (that has been merged to Cocos2D-X at 2016). To start the development, visit the download page, and choose the full one. Start any web server on that directory.

Need more resource? You can watch this awesome youtube playlist (2014 but still quite relevant), or see the website (warning: mostly broken link) or github for more reference.

Or you might want to try, comparison:
Actually I want to write this page as Cocos2D tutorial, but for my current project I need engine that can run under a WebView component, so here's the alternative:
I think QiCiEngine (built on top Phaser.io which built on top PixiJS) is quite awesome, lot's of examples, too bad that WebGL that used by Pixi must use Android Lolipop. There's some people that review those engines (see the recommended videos next ot that video). Also you may want to see TileD to help you edit the stages.
at No comments:

2017-06-01

Alternatives for Web and Mobile App in single codebase

If you are a one-man-army.. um I mean full-stack dev, you'll need to reuse most code as much as possible, a single codebase for all platform instead of rewrite everything in programming language best suited for that platform, that is C++/Java for Android, Swift/Obj-C for iOS, and Javascript for Web.

Other than the popular ReactNative (with react-native-web) and NativeScript (with angular-part that can be reused), there are ReactXP (Microsoft), is a fork of ReactNative (Facebook), with some extras: TypeScript (future proposed ES7 with better tooling), single codebase for all common platform: Android, iOS, Web, and Windows.
To use them, you should clone their repository, and use the provided Hello World.

git clone https://github.com/microsoft/reactxp
cp -r reactxp/samples/hello-world rxp1
cd rxp1
npm install
npm run web-watch # or rn-watch for mobile

Open index.html using browser, preferably use a web server instead of file:// protocol. Install ReactDevTools for better debugging experience. The directories should show something like this:

├── android
│   ├── app
│   ├── build.gradle
│   ├── gradle
│   ├── gradle.properties
│   ├── gradlew
│   ├── gradlew.bat
│   ├── keystores
│   └── settings.gradle
├── index.android.js
├── index.html # <-- the main code for web
├── index.ios.js
├── ios
│   ├── RXPHelloWorld
│   ├── RXPHelloWorldTests
│   └── RXPHelloWorld.xcodeproj
├── node_modules
├── package.json
├── README.md
├── src # <-- your common source code / components here
│   ├── App.tsx 
│   ├── index.tsx
│   ├── ToggleSwitch.tsx
│   └── typings
├── tsconfig.json
└── webpack.config.ts

Now you can edit the App.tsx and start digging, but probably you'll need to wait around end of this year, until it has more features.

Next are the Cordova-based (that would be slower than native since it's browser based, also user on the older devices that has buggy 1 probably need to install last version of CrossWalk), there are:
And there are new one similar to ReactNative that supported by Alibaba (internet giant from China), called Weex that can be used with VueJS, and recently joined Apache Foundation.

Haven't tried them all, but..
  • by skimming the specs and the docs (that i prefer ones that not focusing on mobile-only), 
  • and since I dislike Angular (sorry Google), 
  • not really into React (and whatever state management libraries available: Flux, AltJS, Redux, MobX, Cerebral), 
  • prefer to use RiotJS > VueJS (since I think they take the good parts of Angular and React)
  • not really like write in Typescript (but I like using libraries that written in Typescript ^^ because parameter info/object-property checking is quite helpful on development), 
I think the most prospect is Weex (hey it's native) and Quasar Framework (really cool built in components).

Btw have you heard PWA (youtube)?

Note: this article written in 26th April 2017, scheduled to be published at 1st June 2017, probably things have change since then.

Github Stat Apr 2017 Watch Star Fork
ReactNative 2.9 47.5 11.0
Weex 1.9 14.0 1.9
Ionic 1.7 29.2 7.4
NativeScript 0.6 10.0 0.7
Framework7 0.5 9.5 2.0
OnsenUI 0.2 4.6 0.6
ReactXP 0.2 4.4 0.2
IncubatorWeex 0.1 2.2 0.3
Quasar 0.1 1.9 0.1
Phonon 0.0 0.3 0.0
React 4.4 65.3 12.1
Angular 4.4 55.5 27.7
Vue 2.9 51.5 6.89
Riot 0.4 11.8 0.9
Mithril 0.3 7.5 0.6

Also note that I didn't include other cross platform mobile-only development alternative or ones that specialized for gaming or must-pay, such as Xamarin/Unity3D, Cocos2D, RemObject's Silver, etc
at No comments:

2017-05-26

GotRo Framework Tutorial: Go, Redis and PostgreSQL

Gotro is opinionated colleciton of libraries and framework for Go, it's a rewrite of Gokil framework that specially built for Go+Redis+PostgreSQL web application development. Previously Gokil written using the infamous julienschmidt's httprouter, the fastest on that time (2014), but a year later (2015) forked as buaazp's fasthttprouter that built based on faster valyala's fasthttp. In this tutorial, we will learn how to use this framework with Redis and PostgreSQL to create a SoHo or even medium-enterprise-class web projects.

Each folder on the directory/package uses 1 letter that contains specific commonly used functions, that are:

  • A - Array
  • B - Boolean
  • C - Character (or Rune)
  • D - Database
  • F - Floating Point
  • L - Logging
  • M - Map
  • I - Integer
  • S - String
  • T - Time (and Date)
  • W - Web (the "framework") -- deprecated, use W2 instead
  • X - Anything (aka interface{})
  • Z - Z-Template Engine, that has syntax similar to ruby string interpolation #{foo} with additional other that javascript friendly syntax {/* foo */}, [/* bar */], /*! bar */
To use this libraries, run this command on the console:

go get -u -v github.com/kokizzu/gotro

To start a new project, copy a directory from W/example-simplified folder to your $GOPATH/src, that's the base of your project, that should contain something like this:

├── public
│   └── lib
│       └── jquery.js
├── start_dev.sh
├── server.go
└── views
    ├── error.html
    ├── layout.html
    ├── login_example.html
    ├── named_params_example.html
    ├── post_values_example.html
    └── query_string_example.html

To start the development server, run ./start_dev.sh, it would show something like this:

set ownership of $GOROOT..
remove $GOPATH/pkg if go upgraded/downgraded..
precompile all dependencies..
hello1
starting gin..
[gin] listening on port 3000
2017-05-26 10:55:59.835 StartServer ▶ Gotro Example [DEVELOPMENT] server with 6 route(s) on :3001
  Work Directory: /home/asd/go/src/hello1/

If you got an error, probably because you haven't installed Redis, that being used in this example to store sessions, to do that on Ubuntu, you can type:

sudo apt-get install redis-server
sudo systemctl enable redis-server
sudo systemctl start redis-server

To see the example, open your browser at http://localhost:3000 
Port 3000 is the proxy port for gin program that auto-recompile if the source code changed, the server itself listens on port 3001, if you change it on the source code, you must also change the gin target port on start_dev.sh file, by replacing -a 3001 and -p 3000   

Next we see the example on the server.go file:

redis_conn := Rd.NewRedisSession(``, ``, 9, `session::`)
global_conn := Rd.NewRedisSession(``, ``, 10, `session::`)
W.InitSession(`Aaa`, 2*24*time.Hour, 1*24*time.Hour, *redis_conn, *global_conn)
W.Mailers = map[string]*W.SmtpConfig{
``: {
Name:     `Mailer Daemon`,
Username: `test.test`,
Password: `123456`,
Hostname: `smtp.gmail.com`,
Port:     587,
},
}
W.Assets = ASSETS
W.Webmasters = WEBMASTER_EMAILS
W.Routes = ROUTERS
W.Filters = []W.Action{AuthFilter}
// web engine
server := W.NewEngine(DEBUG_MODE, false, PROJECT_NAME+VERSION, ROOT_DIR)
server.StartServer(LISTEN_ADDR)

There are 2 redis connection, one for storing local session, one for storing global session (used cross app communication).
You must call W.InitSession to tell the framework name of the cookie, default expiration (how long until a cookie expired, and every how long we should renew). On the next line, we set the mailer W.Mailers, connection that we use to send if there are panic or any other critical error within your web server.
W.Assets is the assets file, should contain any css or javascript script that will be included on every page, the assets should be saved on the public/css/ or public/js/ directory. This is the example how to fill them:

 var ASSETS = [][2]string{
//// http://api.jquery.com/ 1.11.1
{`js`, `jquery`},
////// http://hayageek.com/docs/jquery-upload-file.php
{`css`, `uploadfile`},
{`js`, `jquery.form`},
{`js`, `jquery.uploadfile`},
//// https://vuejs.org/v2/guide/ 2.0
{`js`, `vue`},
//// http://momentjs.com/ 2.17.1
{`js`, `moment`},
//// github.com/kokizzu/semantic-ui-daterangepicker
{`css`, `daterangepicker`},
{`js`, `daterangepicker`},
//// http://semantic-ui.com 2.2 // should be below `js` and `css` items
{`/css`, `semantic/semantic`},
{`/js`, `semantic/semantic`},
//// global, helpers, project specific
{`/css`, `global`},
{`/js`, `global`},
}

If you start the type of the file with slash, it means it would locate the file in absolute path starting from public/. Currently only js and css files supported.

Next we must set the W.Webmasters, that is the hardcoded superadmin, one that will be receiving the error emails and could be accessed through ctx.IsWebMaster() that matching the ctx.Session.GetStr(`email`) variable with those values.

Next initialization phase, you must set the route W.Routes, which is used to assign an URL path to a handler function, for example:

var ROUTERS = map[string]W.Action{
``:                            LoginExample,
`login_example`:               LoginExample,
`post_values_example`:         PostValuesExample,
`named_params_example/:test1`: NamedParamsExample,
`query_string_example`:        QueryStringExample,
}

In this example, there are five routes with four different handler function (you can put them on a package, normally you separate them on different package based on access level), on the fourth route we capture the :value as string, that can be anything and can be retrieved by calling ctx.ParamStr(`test1`). Here's some example how to separate the handler based on first segment:

`accounting/acct_payments`:            fAccounting.AcctPayments,
`accounting/acct_invoices`:            fAccounting.AcctInvoices,
`employee/attendance_list`:            fEmployee.AttendanceList,
`employee/business_trip`:              fEmployee.BusinessTrip,
`human_resource/business_trip`:        fHumanResource.BusinessTrip,
`human_resource/employee/profile/:id`: fHumanResource.EmployeeProfileEdit,
`human_resource/employees`:            fHumanResource.Employees,

A handler function should have exactly one parameter with type *W.Context, for example:

func PostValuesExample(ctx *W.Context) {
if ctx.IsAjax() {
ajax := AjaxResponse()
value := ctx.Posts().GetStr(`test2`)
ajax.Set(`test3`, value)
ctx.AppendJson(ajax.SX)
return
}
ctx.Render(`view1`, M.SX{ // <-- locals of the view
`title`: `Post example`,
`map`: M.SI{`test1`:1,`test4`:4},
`arr`: []int{1,2,3,4},
})
}

On above function, we check if the request method is POST or not, if it's so, we assume that it's sent from AJAX, something like this if using jQuery:

var data = {test2: 'foo'};
$.post('', data, function(res) {
alert("Value: " + res.test3);
}).fail(function(xhr, textStatus, errorThrown ) {
alert(textStatus + '\n' + xhr.status);
});

On above javascript snippet, we send to current page through AJAX HTTP POST method, sending a value with key test2 that filled with string foo. The server later will capture it and sending back to client that sending that string as an object with key test3, not that anything you put on it will be converted to JSON. The javascript will retrieve that value through callback (third line on the javascript snippet).

But if client's request is not a POST method, the server will call ctx.Render that will load a file view1.html from view/ directory, if you need to pass anything to that view, put them on a M.SX that is a map with string key and any value type, note that everything you put in this map will be rendered as json. But what's the syntax? This template engine called Z-Template engine, that designed for simplicity and compatibility with javascript syntax, unlike any other template engine, the syntax will not interfere with Javascript IDE's autocomplete feature, here's the example to render values above:

<h1>#{title}</h1>
<h2>#{something that not exists}</h2>
<script>
  var title = '#{title}'; // 'Post example'
  var a_map = {/* map */}; // {"test1":1,"test4":4}
  var an_arr = [/* arr */]; // [1,2,3,4]
</script>

Different from any other template engine, any value given to the Render method that not being used will show a warning, and any key used in the template that not provided in render function will render the key itself (eg: something that not exists).

Wait, in PHP you can retrieve query parameter using $_GET variable, how to do that in this framework?

// this is Go
ctx.QueryParams().GetInt(`theKey`) // equal to $_GET['theKey']

Now back to the handler function, the ctx parameter can be used to control the output, normally when you call Render method, it would also wrap the rendered view with view/layout.html, but if you did not want that, you can call this:

ctx.NoLayout = true
ctx.Buffer.Reset() // to clear rendered things if you already call Render method
ctx.Title = `something` // to set the title, if you use the layout

Layout view have some provided values (locals), that are: title, project_name, assets (the js and css you give on the assets), is_superadmin (if the current logged in person is a webmaster), debug_mode (always true if you didn't update VERSION variable on compile time).

You can see other methods and properties available, you can see them by control-click the W.Context type from your IDE (Gogland, Wide, Visual Studio Code, etc).

Now how to connect to the database? First you must install the database, for example PostgreSQL 9.6 in Ubuntu:

sudo apt-get install postgresql
sudo systemctl enable postgresql
hba=/etc/postgresql/9.6/main/pg_hba.conf
sudo sed -i 's|local   all             all                                     peer|local all all trust|g' $hba
sudo sed -i 's|host    all             all             127.0.0.1/32            md5|host all all 127.0.0.1/32 trust|g' $hba
sudo sed -i 's|host    all             all             ::1/128                 md5|host all all ::1/128 trust|g' $hba
echo 'local all test1 trust' sudo tee -a $hba # if needed
sudo systemctl start postgresql 
sudo su - postgres <<EOF
createuser test1
createdb test1
psql -c 'GRANT ALL PRIVILEGES ON DATABASE test1 TO test1;'

EOF

After testing if your database created correctly, you must create a directory, for example model/ then create a file inside it, for example conn.go with these content:

package model
import (
"github.com/kokizzu/gotro/D/Pg"
_ "github.com/lib/pq"
)
var PG_W, PG_R *Pg.RDBMS
func init() {
PG_W = Pg.NewConn(`test1`, `test1`) 
        // ^ later when scaling we replace this one
PG_R = Pg.NewConn(`test1`, `test1`)
}

On the code above we create 2 connection, writer and reader, this is the recommended way to scale the reader through multiple servers, if you need better writer (but didn't support join, you can use ScyllaDB or Redis). Next, we create a program to initialize our tables, for example in go/init.go:

package main
import "hello1/model"
func main() {
  model.PG_W.CreateBaseTable(`users`, `users`)
  model.PG_W.CreateBaseTable(`todos`, `users`) // 2nd table
}

You must execute the gotro/D/Pg/functions.sql using psql before running the code above, it would create 2 tables with indexes with 2 log tables, triggers and some indexes, you can check it inside psql -U test1 using \dt+ or \d users command, that would show something like this:

                          Table "public.users" 
  Column    |           Type           |                     Modifiers 
------------+--------------------------+-----------------------------------------
id          | bigint                   | not null default nextval('users_id_seq'::regclass) 
unique_id   | character varying(4096)  | 
created_at  | timestamp with time zone | default now() 
updated_at  | timestamp with time zone | 
deleted_at  | timestamp with time zone | 
restored_at | timestamp with time zone | 
modified_at | timestamp with time zone | default now() 
created_by  | bigint                   | 
updated_by  | bigint                   | 
deleted_by  | bigint                   | 
restored_by | bigint                   | 
is_deleted  | boolean                  | default false 
data        | jsonb                    |

This is our generic table, what if we need more columns? You don't need to alter table, we use PostgreSQL's JSONB column data. JSONB is very powerful, it can be indexed, queried using arrow operator, greater than its competitor. Using these exact table design, we can store the old and updated value on the log, everytime somebody changed the value.

Ok, now let's create a real model from users table, create a package and file mUsers/m_users.go with content:

package mUsers
import (
 "Billions/sql"
 "github.com/kokizzu/gotro/A"
 "github.com/kokizzu/gotro/D/Pg"
 "github.com/kokizzu/gotro/I"
 "github.com/kokizzu/gotro/M"
 "github.com/kokizzu/gotro/S"
 "github.com/kokizzu/gotro/T"
 "github.com/kokizzu/gotro/W"
)
const TABLE = `users`
var TM_MASTER Pg.TableModel
var SELECT = ``
var Z func(string) string
var ZZ func(string) string
var ZJ func(string) string
var ZB func(bool) string
var ZI func(int64) string
var ZLIKE func(string) string
var ZT func(...string) string
var PG_W, PG_R *Pg.RDBMS
func init() {
 Z = S.Z
 ZB = S.ZB
 ZZ = S.ZZ
 ZJ = S.ZJ
 ZI = S.ZI
 ZLIKE = S.ZLIKE
 ZT = S.ZT
 PG_W = sql.PG_W
 PG_R = sql.PG_R
 TM_MASTER = Pg.TableModel{
  CacheName: TABLE + `_USERS_MASTER`,
  Fields: []Pg.FieldModel{
   {Key: `id`},
   {Key: `is_deleted`},
   {Key: `modified_at`},
   {Label: `E-Mail(s)`, Key: `emails`, CustomQuery: `emails_join(data)`, Type: `emails`, FormTooltip: `separate with comma`},
   {Label: `Phone`, Key: `phone`, Type: `phone`, FormHide: true},
   {Label: `Full Name`, Key: `full_name`},
  },
 }
 SELECT = TM_MASTER.Select()
}
func One_ByID(id string) M.SX {
 ram_key := ZT(id)
 query := ram_key + `
SELECT ` + SELECT + `
FROM ` + TABLE + ` x1
WHERE x1.id::TEXT = ` + Z(id)
 return PG_R.CQFirstMap(TABLE, ram_key, query)
}
func Search_ByQueryParams(qp *Pg.QueryParams) {
 qp.RamKey = ZT(qp.Term)
 if qp.Term != `` {
  qp.Where += ` AND (x1.data->>'name') LIKE ` + ZLIKE(qp.Term)
 }
 qp.From = `FROM ` + TABLE + ` x1`
 qp.OrderBy = `x1.id`
 qp.Select = SELECT
 qp.SearchQuery_ByConn(PG_W)

 }
/* accessed through: {"order":["-col1","+col2"],"filter":{"is_deleted":false,"created_at":">isodate"},"limit":10,"offset":5}
this will retrieve record 6-15 order by col1 descending, col2 ascending, filtered by is_deleted=false and created_at > isodate
*/

If the example above too complex for you, you can also do manually, see gotro/D/Pg/_example for simpler example. The example above we create a query model, that query from a single table. If you need multiple table (join), you can extend the fields, something like this:

 {Label: `Admin`, Key: `admin`, CustomQuery: `x2.data->>'full_name'`},

And the query params something like this:

qp.From = `FROM ` + TABLE + ` x1 LEFT JOIN ` + mAdmin.TABLE + ` x2 ON (x1.data->>'admin_id') = x2.id::TEXT `

You can also do something like this:

func All_ByStartID_ByLimit_IsAsc_IsIncl(id string, limit int64, is_asc, is_incl bool) A.MSX { sign := S.IfElse(is_asc, `>`, `<`) + S.If(is_incl, `=`) ram_key := ZT(id, I.ToS(limit), sign) where := `` if id != `` { where = `AND x1.id ` + sign + Z(id) } query := ram_key + ` SELECT ` + SELECT + ` FROM ` + TABLE + ` x1 WHERE x1.is_deleted = false ` + where + ` ORDER BY x1.id ` + S.If(!is_asc, `DESC`) + ` LIMIT ` + I.ToS(limit) return PG_R.CQMapArray(table, ram_key, query) } ` // accessed through: {"limit":10} // this will retrieve last 10 records

Or query a single row:

func API_Backoffice_Form(rm *W.RequestModel) { rm.Ajax.SX = One_ByID(rm.Id) } // accessed through: {a:'form',id:'123'} // this will retreive all columns on this record

Or create a save/delete/restore function:

func API_Backoffice_SaveDeleteRestore(rm *W.RequestModel) { PG_W.DoTransaction(func(tx *Pg.Tx) string { dm := Pg.NewRow(tx, TABLE, rm) // NewPostlessData emails := rm.Posts.GetStr(`emails`) // rm is the requestModel, values provided by http req dm.Set_UserEmails(emails) // dm is the dataModel, row we want to update // we can call dm.Get* to retrieve old record values dm.SetStr(`full_name`) dm.UpsertRow() if !rm.Ajax.HasError() { dm.WipeUnwipe(rm.Action) } return rm.Ajax.LastError() }) } // accessed through: {a:'save',full_name:'foo',id:'1'} // update // if without id, it would insert

Then you can call them on a handler or package-internal function, something like:

func API_Backoffice_FormLimit(rm *W.RequestModel) { id := rm.Posts.GetStr(`id`) limit := rm.Posts.GetInt(`limit`) is_asc := rm.Posts.GetBool(`asc`) is_incl := rm.Posts.GetBool(`incl`) result := All_ByStartID_ByLimit_IsAsc_IsIncl(id, limit, is_asc, is_incl) rm.Ajax.Set(`result`, result) } func API_Backoffice_Search(rm *W.RequestModel) { qp := Pg.NewQueryParams(rm.Posts, &TM_MASTER) Search_ByQueryParams(qp) qp.ToMap(rm.Ajax) }

And call those two APIs function inside a handler something like this:

func PrepareVars(ctx *W.Context, title string) { user_id := ctx.Session.GetStr(`id`) rm = &W.RequestModel{ Actor: user_id, DbActor: user_id, Level: ctx.Session.SX, Ctx: ctx, } ctx.Title = title is_ajax := ctx.IsAjax() if is_ajax { rm.Ajax = NewAjaxResponse() } page := rm.Level.GetMSB(`page`) first_segment := ctx.FirstPath() // validate if this user may access this first segment // check their access level, if it's not ok, set rm.Ok to false // then render an error, something like this: /* if is_ajax { rm.Ajax.Error(sql.ERR_403_MUST_LOGIN_HIGHER) ctx.AppendJson(rm.Ajax.SX) return } ctx.Error(403, sql.ERR_403_MUST_LOGIN_HIGHER) return */ if !is_ajax { // render menu based on privilege } else { // prepare variables required for ajax response rm.Posts = ctx.Posts() rm.Action = rm.Posts.GetStr(`a`) id := rm.Posts.GetStr(`id`) rm.Id = S.IfElse(id == `0`, ``, id) } } func Users(ctx *W.Context) { rm := PrepareVars(ctx, `Users`) if !rm.Ok { return } if rm.IsAjax() { // handle ajax switch rm.Action { case `search`: // @API mUsers.API_Backoffice_Search(rm) case `form_limit`: // @API mUsers.API_Backoffice_FormLimit(rm) case `form`: // @API mUsers.API_Backoffice_Form(rm) case `save`, `delete`, `restore`: // @ffPI mUsers.API_Backoffice_SaveDeleteRestore(rm) default: // @API-END handler.ErrorHandler(rm.Ajax, rm.Action) } ctx.AppendJson(rm.Ajax.SX) return } locals := W.Ajax{SX: M.SX{ `title`: ctx.Title, }} qp := Pg.NewQueryParams(nil, &mUsers.TM_MASTER) mUsers.Search_ByQueryParams(qp) qp.ToMap(locals) ctx.Render(`backoffice/users`, locals.SX) }

Now that we're done creating the backend API server, all that's left is create the systemd service hello1.service:

[Unit] Description=My Hello1 Service After=network-online.target postgresql.service Wants=network-online.target systemd-networkd-wait-online.service [Service] Type=simple Restart=on-failure User=yourusername Group=users WorkingDirectory=/home/yourusername/web ExecStart=/home/yourusername/web/run_production.sh ExecStop=/usr/bin/killall Hello1 LimitNOFILE=2097152 LimitNPROC=65536 ProtectSystem=full NoNewPrivileges=true [Install] WantedBy=multi-user.target

Create the run_production.sh shell script

#!/usr/bin/env bash ofile=logs/access_`date +%F_%H%M%S`.log echo Logging into: `pwd`/$ofile unbuffer time ./Hello1 | tee $ofile

Then compile the binary (you: can also set the VERSION here, to make it production):

go build -ldflags " -X main.LISTEN_ADDR=:${SUB_PORT} " -o /tmp/Subscriber

Copy the binary, the script above, and whole public/ and views/ directory to the server /home/yourusername/web, copy the service file to the /usr/lib/systemd/system/ then reload the systemd service on the server:

sudo systemctl daemon-reload sudo systemctl enable hello1 sudo systemctl start hello1

you're good to go, you can check the service status using journalctl -f hello1
of course you can automate the hassle above using scp or rsync command.

Well, that's all for now, you can see the complete example on W/example-complex directory, if you have any question, you can contact me through telegram @kokizzu, for frontend stuff, I recommend to learn about VueJS or Weex for mobile.
at No comments:

2017-05-22

Go-Redis vs RediGo (also Aerospike)

This is an old benchmark result that test Redis and Aerospike, both are in-memory database, I did this about December last year, that I used to test Redis agains Aerospike for cases of storing random session per request:

 Redis (redigo)
Transactions:                  52343 hits 
Availability:                 100.00 % 
Elapsed time:                   9.16 secs 
Data transferred:              17.02 MB 
Response time:                  0.04 secs 
Transaction rate:            5714.30 trans/sec  (1654 worst >1M uQ sess)
Throughput:                     1.86 MB/sec 
Concurrency:                  252.55 
Successful transactions:       52343 
Failed transactions:               0 
Longest transaction:            1.13 
Shortest transaction:           0.00

 Aerospike (aerospike-client-go)
Transactions:                  80806 hits
Availability:                 100.00 %
Elapsed time:                   9.71 secs
Data transferred:              26.28 MB
Response time:                  0.03 secs
Transaction rate:            8321.94 trans/sec (8999 best, 7769 worst)
Throughput:                     2.71 MB/sec
Concurrency:                  251.91
Successful transactions:       80806
Failed transactions:               0
Longest transaction:            1.17
Shortest transaction:           0.00

 Redis (go-redis)
Transactions:                  91187 hits 
Availability:                 100.00 % 
Elapsed time:                   9.95 secs 
Data transferred:              29.65 MB 
Response time:                  0.03 secs 
Transaction rate:            9164.52 trans/sec (3536 worst >1M uQ sess)
Throughput:                     2.98 MB/sec 
Concurrency:                  252.70 
Successful transactions:       91187 
Failed transactions:               0 
Longest transaction:            0.20 
Shortest transaction:           0.00 

The bad part about Redis (that uses SkipList), more data we store, it slows down faster, in this case 1 million sessions stored slows Redis down by more than 60%, while Aerospike only slowed down by 10%).
at No comments:

Redis 3.2.1 vs ScyllaDB 1.7RC2

Since Scylla still have no secondary index, all I can do to use Scylla is to replace Redis for storing user login sessions, this benchmark only test read queries (queries that always returns zero request because the record does not exists) and the result are:

# Redis 
$ hey -c 255 -n 255000 http://localhost:3001
3089 requests done.
7217 requests done.
11691 requests done.
*snip*
241822 requests done.
246305 requests done.
250697 requests done.
All requests done.

 Summary:
  Total:        29.5162 secs
  Slowest:      0.1647 secs
  Fastest:      0.0003 secs
  Average:      0.0294 secs
  Requests/sec: 8639.3205
  Total data:   2732835000 bytes
  Size/request: 10717 bytes

 Status code distribution:
  [200] 255000 responses

 Response time histogram:
  0.000 [1]     |
  0.017 [4084]  |∎
  0.033 [194502]|∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  0.050 [54311] |∎∎∎∎∎∎∎∎∎∎∎
  0.066 [1812]  |
  0.082 [65]    |
  0.099 [7]     |
  0.115 [122]   |
  0.132 [48]    |
  0.148 [25]    |
  0.165 [23]    |

 Latency distribution:
  10% in 0.0231 secs
  25% in 0.0255 secs
  50% in 0.0286 secs
  75% in 0.0325 secs
  90% in 0.0370 secs
  95% in 0.0404 secs
  99% in 0.0487 secs

 # ScyllaDB best response time
$ hey -c 255 -n 255000 http://localhost:3001
2114 requests done.
4874 requests done.
7714 requests done.
*snip*
247202 requests done.
249898 requests done.
252610 requests done.
All requests done.

 Summary:
  Total:        48.5436 secs
  Slowest:      0.2649 secs
  Fastest:      0.0013 secs
  Average:      0.0483 secs
  Requests/sec: 5253.0127
  Total data:   2732835000 bytes
  Size/request: 10717 bytes

 Status code distribution:
  [200] 255000 responses

 Response time histogram:
  0.001 [1]     |
  0.028 [6804]  |∎∎
  0.054 [176673]|∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  0.080 [66748] |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  0.107 [3728]  |∎
  0.133 [470]   |
  0.159 [250]   |
  0.186 [46]    |
  0.212 [79]    |
  0.239 [144]   |
  0.265 [57]    |

 Latency distribution:
  10% in 0.0334 secs
  25% in 0.0399 secs
  50% in 0.0466 secs
  75% in 0.0552 secs
  90% in 0.0636 secs
  95% in 0.0699 secs

  99% in 0.0899 secs

 # ScyllaDB best Req/s
$ hey -c 255 -n 255000 http://localhost:3001
2188 requests done.
4910 requests done.
7019 requests done.
*snip*
244547 requests done.
249813 requests done.
254894 requests done.
All requests done.

 Summary:
  Total:        42.0725 secs
  Slowest:      8.0907 secs
  Fastest:      0.0002 secs
  Average:      0.0418 secs
  Requests/sec: 6060.9647
  Total data:   2732835000 bytes
  Size/request: 10717 bytes

 Status code distribution:
  [200] 255000 responses

 Response time histogram:
  0.000 [1]     |
  0.809 [254744]|∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  1.618 [0]     |
  2.427 [0]     |
  3.236 [0]     |
  4.045 [0]     |
  4.854 [0]     |
  5.664 [0]     |
  6.473 [0]     |
  7.282 [0]     |
  8.091 [255]   |

 Latency distribution:
  10% in 0.0069 secs
  25% in 0.0183 secs
  50% in 0.0347 secs
  75% in 0.0470 secs
  90% in 0.0573 secs
  95% in 0.0640 secs
  99% in 0.0843 secs

This is the example code of the main function using gotro framework that used to do this benchmark:

//// Testing Redis:
//login_conn := Rd.NewRedisSession(``, ``, 1, `session::`)
//global_conn := Rd.NewRedisSession(``, ``, 3, `global::`)

 // Testing Scylla:
login_conn:= Sc.NewScyllaSession(`127.0.0.1`, `session`, `login`, ``, ``)
global_conn := Sc.NewScyllaSession(`127.0.0.1`, `session`, `global`, ``, ``)

 // see example
W.InitSession(`SK`, 12*time.Hour, 6*time.Hour, *login_conn, *global_conn)
W.Mailers = ...
W.Assets = ...
W.Webmasters = ...
W.Routes = ...
server := W.NewEngine(DEBUG_MODE, false, `test`, ROOT_DIR)
server.StartServer(LISTEN_ADDR)

And just like before, after doing this intensive benchmark, the Ubuntu showed an error for Scylla:


The bad part about Redis is the scalability stuck on single core, if you add more server the write will not scale, so Scylla is better replacement if you want to do horizontal scaling.
at No comments: