Create a Long running Kubernetes probe for debugging cluster issues

Everyone will say this is a bad idea. I say, this is a good idea. And you'll know it when you need it :)

Probe Pod

Kubernetes pods are not meant to be running without doing anything. They are not your desktop machines.

But sometimes, you may need such a pod to inspect or debug something. Basically a simple pod that is just there in the cluster so that you can exec into it and debug stuffs within the cluster (e.g. network connections, db connections etc.)

Building a Probe Pod

Create a Dockerfile

# Start with a minimal version of linux
FROM alpine

# Load it with the tools that you may need. 
# For example, I need PostgreSQL client
RUN apk --no-cache add postgresql14-client

# Run an infinite loop so that the pod does not die
CMD ["/bin/sh", "-c", "--", "while true; do sleep 30; done;"]

Once the Dockerfile is there, you can build an image like this -

docker build --platform=linux/amd64 -t aksmtr/probe:latest-amd64 .
docker build --platform=linux/arm64 -t aksmtr/probe:latest-arm64 .

Please note, you may need to specify the platform tag if you are building it in a platform that is different than the target platform. In my case, I was building it in MacBook M1 Air (arm64), and intended to run it in Linux (amd64).

Push to Docker Hub

Login to Docker 

docker login -u "<your docker username>" -p "<password>" docker.io

Next, push the changes -

docker push aksmtr/probe:latest-amd64
docker push aksmtr/probe:latest-arm64

Using a Probe Pod

Kubernetes Deployment 

Next you will use a Kubernetes Deployment to use this.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: probe
  namespace: default
  labels:
    app: probe
spec:
  replicas: 1
  selector:
    matchLabels:
      app: probe
  template:
    metadata:
      labels:
        app: probe
    spec:
      containers:
        - name: probe
          image: docker.io/<your username>/probe:latest-amd64

You can run the deployment -

kubectl apply -f probe.yml

Then you can log in to the probe -

kubectl exec -it <pod> -- sh

That's it.