Troubleshooting Cheatsheet when Debugging Service in Kubernetes

Gain Shell Access to Pod (similar to ssh)

kubectl exec -it <pod> -- bash

This only working if the pod contains bash. Change bash to sh if bash not found.

Temporary Spawn Ubuntu Pod

Sometimes it is impossible to gain shell access to the existing running pod. The reason maybe: no shell binary in the pod or even we don't have no access to the pod.
The other way that we can try is spawning a pod temporarily then we can use this pod to debug the running service.

kubectl run -i \
	--tty busybox \
	--image=ubuntu \
	--restart=Never \
	-- bash

In that pod, you can use apt to install common utilities to debug service:

apt update -y
apt install -y dnsutils curl netcat

After finish, don't forget to delete the pod

kubectl delete pod busybox

Useful Commands

# check the dns resolution (you can try to lookup kubernetes service host as well here)
nslookup <domain>

# check whether the host:port is open
nc -zv <host> <port>