Pyroscope: Continuous Tracing in Go, Python, or Ruby

Recently I stumbled upon slow library/function problem and don't know chich part that causes it, and found out that there's a easy way to trace either Go, Ruby, or Python code using Pyroscope. The feature is a bit minimalist, there's no memory usage tracing yet unlike in gops or pprof. Pyroscope consist 2 parts: the server and the agent/client library (if using Golang) or executor (if using Ruby or Python). Here's the way how to run and start Pyroscope server:

# run server using docker

docker run -it -p 4040:4040 pyroscope/pyroscope:latest server

And here's the example on how to use the client library/agent (modifying Go's source code, just like in DataDog or any other APM tools) and install  the Pyroscope CLI to run Ruby/Python scripts:

# golang, add agent inside the source code

import "github.com/pyroscope-io/pyroscope/pkg/agent/profiler"

func main() {

  profiler.Start(profiler.Config{

    ApplicationName: "my.app.server", 

    ServerAddress:   "http://pyroscope:4040",

  })

  // rest of your code

}

# ruby or python, install CLI client 

cd /tmp

wget https://dl.pyroscope.io/release/pyroscope_0.0.28_amd64.deb

sudo apt-get install ./pyroscope_0.0.28_amd64.deb

# ruby

pyroscope exec ruby yourcode.rb

# python

pyroscope exec python yourcode.py

It would show something like this if you open the server URL (localhost:4040) in the browser, so you can check which part of the code that took most of the runtime.

GOPS: Trace your Golang service with ease

GoPS is one alternative (also made by Google, other than pprof) to measure, trace or diagnose the performance and memory usage your Go-powered service/long-lived program. The usage is very easy, you just need to import and add 3 lines in your main (so the gops command line can communicate with your program):

import "github.com/google/gops/agent"

func main() {

  if err := agent.Listen(agent.Options{}); err != nil {

    log.Fatal(err)

  }

  // remaining of your long-lived program logic

}

If you don't put those lines, you can still use gops limited to get list of programs running on your computer/server that made with Go with limited statistics information, using these commands:

$ go get -u -v github.com/google/gops

$ gops  # show the list of running golang program

1248    1       dnscrypt-proxy  go1.13.4  /usr/bin/dnscrypt-proxy

1259    1       containerd      go1.13.15 /usr/bin/containerd

18220   1       dockerd         go1.13.15 /usr/bin/dockerd

1342132 1306434 docker          go1.13.15 /usr/bin/docker

$ gops tree # show running process in tree

$ gops PID # check the stats and whether the program have GOPS agent

#########################################################

# these commands below only available
# if the binary compiled with GOPS agent

# PID can be replaced with GOPS host:port of that program

$ gops stack PID # get current stack trace of running PID

$ gops memstats PID # get memory statistics of running PID

$ gops gc PID # force garbage collection

$ gops setgc PID X # set GC percentage

$ gops pprof-cpu PID # get cpu profile graph

$ gops pprof-heap PID # get memory usage profile graph

profile saved at /tmp/heap_profile070676630

$ gops trace PID # get 5 sec execution trace

# you can install graphviz to visualize the cpu/memory profile

$ sudo apt install graphviz

# visualize the cpu/memory profile graph on the web browser

$ go tool pprof /tmp/heap_profile070676630

> web 

Next step is analyze the call graph for the memory leaks (which mostly just wrongly/forgot to defer body/sql rows or holding slice reference of huge buffer or certain framework's cache trashing) or slow functions, whichever your mission are.

What if golang service you need to trace it inside Kubernetes pod that the GOPS address (host:port) not exposed to outside-world? Kubernetes is a popular solution for companies that manages bunch of servers/microservices or cloud like (GKE, AKS, Amazon EKS, ACK, DOKS, etc) but obviously overkill solution for small companies that doesn't need to scale elastically (or the servers are less than 10 or not using microservice architecture).

First, you must compile gops statically so it can be run inside alpine container (which mostly what people use):

$ cd $GOPATH/go/src/github.com/google/gops

$ export CGO_ENABLED=0

$ go mod vendor

$ go build

# copy gops to your kubernetes pod

$ export POD_NAME=blabla

$ kubectl cp ./gops $POD_NAME:/bin

# ssh/exec to your pod

$ kubectl exec -it $POD_NAME -- sh

$ gops

# for example you want to check heap profile for PID=1

$ gops pprof-heap 1

$ exit

# copy back trace file to local, then you can analyze the dump

kubectl cp $POD:/tmp/heap_profile070676630 out.dump

But if your address and port are exposed you can directly use gops from your computer to the pod or create a tunnel inside the pod if it doesn't have public IP, for example using ngrok.

Btw if you know any companies migrating from/to certain language (especially Go), frameworks or database, you can contribute here.