Profiling Go Service remotely in kubernetes

Profiling a service in Go

pprof helps in collecting different type of performance profiles in Go, such as CPU, Heap, Goroutine etc. How to setup an application to generate these profiles is described at official go language documentation.

Making sense of Profile

To make sense of a profile, we can use go tool chain. For example, if we have a CPU profile, we can analyze it using go tool pprof -http localhost:9000 cpu.profile. This will open profile information in a browser and we can analyze it.

What if pod is not exposed as a service

If service is reachable and you can ssh to it, it is easy to create a ssh tunnel and analyze the profile from local machine itself. But there are cases when application is running in kubernetes and we can’t reach it using tunnel. In this scenario, we can get profile on node as then copy that profile to a machine that has go installed. Here are the steps to do that:

1. Make sure pprof is enabled for application
2. Choose a pod to get the profile from
3. Exec to chosen pod using kubectl exec -it <pod name> — sh
4. Once in pod run curl localhost:9000/debug/pprof/goroutine —output goroutine-file
5. If file is created and saved, exit from pod
6. From your local machine, run kubectl cp <remote-pod-name>:<location-to-file> <local file location>/file-name
7. Run go tool pprof -http localhost:9000 <local file location>/file-name to analyze profile in browser