Live Debugging

Debugging Kubernetes Deployments

This article covers some of the general guidelines you can use to debug your Kubernetes deployments and some of the common errors and issues you can expect to encounter.
Serkan Ozal
15 mins read

Kubernetes is a useful tool for deploying and managing containerized applications, yet even seasoned Kubernetes enthusiasts agree that it’s hard to debug Kubernetes deployments and failing pods. This is because of the distributed nature of Kubernetes, which makes it hard to reproduce the exact issue and determine the root cause.

This article will cover some of the general guidelines you can use to debug your Kubernetes deployments and some of the common errors and issues you can expect to encounter.

Additionally, Scott Surovich and Marc Boorshtein help you learn about Kubernetes cluster deployment and management and discover how to deploy an entire platform to the cloud using continuous integration and continuous delivery (CI/CD) in their book Kubernetes and Docker - An Enterprise Guide.

Tools to Use to Debug the Kubernetes Cluster

It’s important to ensure that our process remains the same whether we’re debugging our application in Kubernetes or a physical machine. The tools we use will be the same, but with Kubernetes we're going to probe the state and outputs of the system. We can always start our debugging process using kubectl, or we can use some of the common Kubernetes debugging tools.

Kubectl is a command-line tool that lets you control your Kubernetes cluster and comes with a wide range of commands, such as kubectl get [-o wide] nodes, pods, and svc.One of the critical commands is get, which lists one or more resources. For example, kubectl get pods will help you check your pod status, and you can check the status of worker nodes using kubectl get nodes. You can also pass -o with an output format to get additional information. For example, when you pass -o wide with kubectl get pods, the node is included in the output (kubectl get pods -o wide).

Note: One of the important flags that come with the kubectl command is -w(watch flag), which lets you start watching the updates to a particular object (for example, kubectl get pods -w):


kubectl describe (node, pod, svc) <name> -o yaml

The kubectl describe nodes display detailed information about resources. For example, the kubectl describe nodes <node name> displays detailed information about the node with the name <node name>. Similarly, to get detailed information about pod use, you should use kubectl describe pod <pod name>. If you pass the flag -o yaml at the end of the kubectl describe command, it displays the output in yaml format (kubectl describe pod <pod name> -o yaml).

kubectl logs

Kubectl logs is another useful debugging command used to print the logs for a container in a pod. For example, kubectl logs <pod name> prints the pod's logs with a name pod name.

Note: To stream the logs, use the -f flag along with the kubectl logs command. The -f flag works similarly to the tail -f command in Linux. An example is kubectl logs -f <pod name>.

kubectl -v(verbosity)

For kubectl, verbosity is controlled with -v or --v flags. After -v, we will pass an integer to represent the log level. The integer ranges from 0 to 9, where 0 is least verbose and 9 is most verbose. An example is kubectl -v=9 get nodes display HTTP request contents without truncation of contents.

kubectl exec

Kubectl exec is a useful command to debug the running container. You can run commands like kubectl exec <pod name> -- cat /var/log/messages to look at the logs from a given pod, or kubectl exec -it <pod name> --sh to log in to the given pod.

kubectl get events

Kubectl events give you high-level information about what is happening inside the cluster. To list all the events, you can use a command like kubectl get events, or if you are looking for a specific type of event such as Warning, you can use kubectl get events --field-selector type=Warning.

Debugging Pod

There are two common reasons why pods fail in Kubernetes:

  • Startup failure: A container inside the pod doesn’t start.
  • Runtime failure: The application code fails after container startup.

Debugging Common Pod Errors with Step-by-Step and Real-World Examples

CrashLoopBackoff

A CrashLoopBackoff error means that when your pod starts, it crashes; it then tries to start again, but crashes again. Here are some of the common causes of CrashLoopBackoff:

  • An error in the application inside the container.
  • Misconfiguration inside the container (such as a missing ENTRYPOINT or CMD).
  • A liveness probe that failed too many times.
Commands to Run to Identify the Issue

The first step in debugging this issue is to check pod status by running the kubectl get pods command.

$ kubectl get podsNAME        READY     STATUS              RESTARTS    AGEbusybox     0/1       CrashLoopBackOff      1           27s

Here we see our pod status under the STATUS column and verify that it’s in a CrashLoopBackOff state.

As a next troubleshooting step, we are going to run the kubectl logs command to print the logs for a container in a pod.

$ kubectl logs busyboxsleep: invalid number '-z'

Here we see that the reason for this broken pod is specification of an unknown option -z in the sleep command. Verify your pod definition file.


$ cat podbroken.yml
apiVersion: v1
kind: Pod
metadata:
  name: busybox
  labels:
    env: Prod
spec:
  containers:
  - name: busybox
    image: busybox
    args:
      - "sleep"
      - "-z"

To fix this issue, you’ll need to specify a valid option under the sleep command in your pod definition file and then run the following command

$ kubectl create -f broken.ymlpod/busybox created

Verify the status of your pod.

$ kubectl get podsNAME        READY     STATUS              RESTARTS    AGEbusybox     1/1       Running               1           63s

ErrImagePull/ImagePullBackOff

An ErrImagePull/ImagePullBackOff error occurs when you are not able to pull the desired docker image. These are some of the common causes of this error:

  • The image you have provided has an invalid name.
  • The image tag doesn’t exist.
  • The specified image is in the private registry.
Commands to Run to Identify the Issue

As with the CrashLoopBackoff error, our first troubleshooting step starts with getting the pod status using the kubectl get pods command.

Next we will run the kubectl describe command to get detailed information about the pod.

$ kubectl describe pod redisEvents:  Type      Reason      Age                 From                Message----      ------      ----                ----                -------  Normal    Scheduled   46s                 default-scheduler   Successfully assigned default/redis to node01  Normal    BackOff     16s (x2 over 43s)   kubelet, node01     Back-off pulling image "redis123"  Warning   Failed      16s (x2 over 43s)   kubelet, node01     Error: ImagePullBackOff  Normal    Pulling     2s (x3 over 45s)    kubelet, node01     Pulling image "redis123"  Warning   Failed      1s (x3 over 44s)    kubelet, node01     Failed to pull image "redis123": rpc error: code = Unknown desc = Error response from daemon: pull access denied for redis123, repository does not exist or may require 'docker login': denied: requested access to the resource is denied  Warning   Failed      1s (x3 over 44s)    kubelet, node01     Error: ErrImagePull

As you can see in the output of the kubectl describe command, it was unable to pull the image name redis123.

To find the correct image name, you can either go to Docker Hub or run the docker search command specifying the image name on the command line.

$ docker search redisNAME                             DESCRIPTION                                     STARS     OFFICIAL   AUTOMATEDredis                            Redis is an open source key-value store that…   8985      [OK]

To fix this issue, we can follow either one of the following approaches:

  1. Start with the kubectl edit command, which allows you to directly edit any API resource you have retrieved via the command-line tool. Go under the spec section and change the image name from redis123 to redis.
  2. If you already have the yaml file, you can edit it directly, and if you don’t, you can use the --dry-run option to generate it.

kubectl edit pod redis...spec:  containers:  - image: redis...
$ kubectl  run redis --image redis --dry-run=client -o yaml > redispod.yml

$ cat redispod.yml
apiVersion: v1
kind: Pod
metadata:
 creationTimestamp: null
 labels:
   run: redis
   name: redis
spec:
 containers:
 - image: redis
   name: redis123
   resources: {}
 dnsPolicy: ClusterFirst
 restartPolicy: Always
status: {}

Similarly, under image spec, change the image name to redis and recreate the pod.

$ kubectl create -f redispod.ymlpod/redis created

CreateContainerConfigError

This error usually occurs when you’re missing a ConfigMap or Secret. Both are ways to inject data into the container when it starts up.

Commands to Run to Identify the Issue

We will start our debugging process with the standard Kubernetes debugging command kubectl get pods.

$ kubectl get podsNAME              READY     STATUS                        RESTARTS    AGEconfigmap-pod     0/1       CreateContainerConfigError    0           58s

As the output of kubectl get pods shows, pod is in a CreateContainerConfigError state. Our next command is kubectl describe, which gets the detailed information about the pod.

$ kubectl describe pod configmap-podWarning  Failed       44s (x8 over 2m10s)     kubelet       Error: configmap "my-config" not foundNormal   Pulled       44s                     kubelet       Successfully pulled image "gcr.io/google_containers/busybox" in 1.002510873sNormal   Pulling      31s (x9 over 2m14s)     kubelet       Pulling image "gcr.io/google_containers/busybox"

To retrieve information about the ConfigMap, use this command:

$ kubectl get configmap

As the output of the above command is null, the next step is to verify the pod definition file and create the ConfigMa

apiVersion: v1kind: Podmetadata:  name: configmap-podspec:  containers:    - name: mytest-container      image: busybox      command: [ "/bin/sh", "-c", "env" ]      env:        - name: MY_KEY          valueFrom:            configMapKeyRef:              name: my-config              key: nameTo fix the error, create a ConfigMap file, whose content will look like this:$ cat my-config.yamlapiVersion: v1data:  name: test  value: userkind: ConfigMapmetadata:  name: my-config$ kubectl apply -f configmap-special-config.yamlconfigmap/special-config createdNow run kubectl get configmap to retrieve information about the ConfigMap, and this time, you’ll see the newly created ConfigMap.$ kubectl get configmapNAME              DATA    AGEmy-config         2       11sVerify the status of the pod, and you will see the pod will be running state now.$ kubectl get podsNAME              READY     STATUS        RESTARTS    AGEconfigmap-pod     1/1       Running       4           7m29s

ContainerCreating Pod

The main cause of the ContainerCreating error is that your container is looking for a secret that is not present. Secrets in Kubernetes let you store sensitive information such as tokens, passwords, and ssh keys.

Commands to Run to Identify the Issue

We will start our debugging process with the standard Kubernetes debugging command kubectl get pods.

$ kubectl get podsNAME              READY     STATUS                RESTARTS    AGEsecret-pod        0/1       ContainerCreating     0           7s

As the output of kubectl get pods shows, the pod is in a ContainerCreating state. Our next command is the kubectl describe command, which gets detailed information about the pod.

$ kubectl describe pod secret-podEvents:  Type      Reason      Age                 From                Message----      ------      ----                ----                -------  Normal    Scheduled   25s                 default-scheduler   Successfully assigned default/secret-pod to 17e9458eef1c.mylabserver.com  Normal    Pulled      18s                 kubelet             Successfully pulled image "nginx" in 666.826881ms  Normal    Pulled      17s                 kubelet             Successfully pulled image "nginx" in 472.277634ms  Normal    Pulling     5s (x3 over 18s)    kubelet             Pulling image "nginx"  Warning   Failed      4s (x3 over 17s)    kubelet             Error: secret "myothersecret" not found  Normal    Pulled      4s                  kubelet             Successfully pulled image "nginx" in 476.69613ms

To retrieve information about the secret, use this command:

$ kubectl get secret

As the above command's output is null, the next step is to verify the pod definition file and create the secret.

$ cat secret.ymlapiVersion: v1kind: Podmetadata:  name: secret-podspec:  containers:    - name: test-contaner      image: nginx      envFrom:      - secretRef:          name: myothersecret

To fix the error, create a secret file whose content looks like this.

$ cat secret-data.ymlapiVersion: v1kind: Secretmetadata:  name: myothersecretdata:  USER_NAME: YWRtaW4=  PASSWORD: MWYyZDFlMmU2N2Rm

Secrets are similar to ConfigMap, except they are stored in a base64 encoded or hashed format.

For example, you can use the base64 command to encode any text data.

$ echo -n "username" | base64dXNlcm5hbWU=

Use the following commands to decode the text and print the original text.

$ echo -n "dXNlcm5hbWU=" | base64 --decodeusername$ kubectl create -f secret-data.ymlsecret/myothersecret created

Now run kubectl get secret to retrieve information about a secret, and this time you will see the newly created secret.

$ kubectl get secretNAME              TYPE      DATA    AGEmyothersecret     Opaque    2       20m

Verify the status of the pod, and you will see that the pod is in running state now.

$ kubectl get podsNAME          READY     STATUS      RESTARTS    AGEsecret-pod    1/1       Running     0           2m36s

Debugging Worker Nodes

A worker node in the Kubernetes cluster is responsible for running your containerized application. To debug worker node failure, we need to follow the same systematic approach and tools as we used while debugging pod failures.To reinforce the concept, we will look at three different scenarios where you see your worker node is in NotReady state.

NotReady State

There may be multiple reasons why your worker node goes into NotReady state. Here are some of them:

  • A virtual machine where the worker node is running shut down.
  • There is a network issue between worker and master node.
  • There is a crash within the Kubernetes software.

But as you know, the kubelet is the binary present on the worker node, and it is responsible for running containers on the node. We’ll start our debugging with a kubelet.

Scenario 1: Worker Node is in NotReady State (kubelet is in inactive [dead] state)

Commands to Run to Identify the Issue

Before we start debugging at the kubelet binary, let’s first check the status of worker nodes using kubectl get nodes.

$ kubectl get nodesNAME            STATUS      ROLES     AGE     VERSIONcontrolplane    Ready       master    21m     v1.19.0node01          NotReady    <none>    20m     v1.19.0

Now, as we have confirmed that our worker node (node01) is in NotReady state, the next command we will run is ps to check the status of the kubelet.

$ ps -ef | grep -i kubeletroot    21200 16471  0 04:46 pts/1    00:00:00 grep -i kubelet

As we see, the kubelet process is not running. We can run the systemctl command to verify it further.

# systemctl status kubelet  kubelet.service - kubelet: The Kubernetes Node Agent    Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)   Drop-In: /etc/systemd/system/kubelet.service.d            └─10-kubeadm.conf    Active: inactive (dead) since Tue 2020-10-20 04:34:33 UTC; 1min 53s ago      Docs: https://kubernetes.io/docs/home/   Process: 1455 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS  Main PID: 1455 (code=exited, status=0/SUCCESS)Oct 20 04:34:33 node01 systemd[1]: Stopping kubelet: The Kubernetes Node Agent...Oct 20 04:34:33 node01 systemd[1]: Stopped kubelet: The Kubernetes Node Agent.Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.

Using the systemctl command, we confirmed that the kubelet is not running (Active: inactive (dead)). Before we debug it further, we can try to start the kubelet service and see if that helps. To start the kubelet, run the systemctl start kubelet command.

# systemctl start kubelet

Verify the status of kubelet again.

# systemctl status kubelet  kubelet.service - kubelet: The Kubernetes Node Agent    Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)   Drop-In: /etc/systemd/system/kubelet.service.d            └─10-kubeadm.conf    Active: active (running) since Tue 2020-10-20 04:37:14 UTC; 1s ago      Docs: https://kubernetes.io/docs/home/  Main PID: 13240 (kubelet)     Tasks: 8 (limit: 4678)    CGroup: /system.slice/kubelet.service            └─13240 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kube

Based on the systemctl status kubelet output, we can say that starting kubelet helped, and now it’s in running state. Let’s verify it from the Kubernetes master node.

$ kubectl get nodesNAME            STATUS      ROLES     AGE   VERSIONcontrolplane    Ready       master    23m   v1.19.0node01          Ready       <none>    23m   v1.19.0

As you can see, the worker node is now in Ready state.

Scenario 2: Worker Node is in NotReady state (kubelet is in activating [auto-restart] state)

In scenario 2, your worker node is again in NotReady state, and you’ve followed all the steps to start the kubelet (systemctl start kubelet), but that doesn’t help. This time, the kubelet service status is activating (auto-restart), which means it’s not able to start.

# systemctl status kubelet  kubelet.service - kubelet: The Kubernetes Node Agent    Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)  Drop-In: /etc/systemd/system/kubelet.service.d           └─10-kubeadm.conf   Active: activating (auto-restart) (Result: exit-code) since Tue 2020-10-20 04:41:58 UTC; 1s ago

Here we’ll start our debugging with a command called journalctl. Using journalctl, you can view the logs collected by systemd. In this case, we will pass the -u option to see the logs of a particular service, kubelet.

# journalctl -u kubelet

You will see the following error message:

Oct 20 04:54:47 node01 kubelet[28692]: F1020 04:54:47.021580   28692 server.go:253] unable to load client CA file /etc/kubernetes/pki/my-test-file.crt: open /etc/kubernetes/pki/my-test-file.crtOct 20 04:54:47 node01 kubelet[28692]: goroutine 1 [running]:

Note: To reach the bottom of the command, press SHIFT+G.

Our next step is to identify the configuration file. Since this scenario is created by kubeadm, you will see the custom configuration file.

# cd /etc/systemd/system/kubelet.service.d/$ cat 10-kubeadm.confEnvironment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf--kubeconfig=/etc/kubernetes/kubelet.conf"Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"

This file also has the reference to the actual configuration used by kubelet. Open this file and check the line starting with clientCAFile:

# cat /var/lib/kubelet/config.yamlapiVersion: kubelet.config.k8s.io/v1beta1authentication:  anonymous:    enabled: false  webhook:    cacheTTL: 0s    enabled: true  x509:    clientCAFile: /etc/kubernetes/pki/my-test-file.crt

You will see the incorrect CA path for clients (/etc/kubernetes/pki/my-test-file.crt. Let’s check the valid path under /etc/kubernetes/pki.

# ls -l /etc/kubernetes/pki/total 4-rw-r--r-- 1 root root 1066 Oct 20 04:14 ca.crt

Replace it with the correct file path and restart the daemon.

clientCAFile: /etc/kubernetes/pki/my-test-file.crt

TO

clientCAFile: /etc/kubernetes/pki/ca.crt# systemctl restart kubelet# systemctl status kubelet  kubelet.service - kubelet: The Kubernetes Node Agent    Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)   Drop-In: /etc/systemd/system/kubelet.service.d            └─10-kubeadm.conf    Active: active (running) since Tue 2020-10-20 05:05:38 UTC; 7s ago

Again, verify it from the Kubernetes master node.

$ kubectl get nodesNAME            STATUS      ROLES     AGE     VERSIONcontrolplane    Ready       master    23m     v1.19.0node01          Ready       <none>    23m     v1.19.0

As you can see, the worker node is now in Ready state.

Scenario 3 Worker Node is in NotReady state (kubelet is in active [running] state)

In scenario 3, your worker node is again in NotReady state, and although you’ve followed all the steps for starting the kubelet (systemctl start kubelet), that doesn’t help. This time, the kubelet service status is loaded, which means that systemctl reads the unit from disk into memory, but it’s not able to start. We will start our debugging with the standard process by running the systemctl status command.

# systemctl status kubelet  kubelet.service - kubelet: The Kubernetes Node Agent    Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)   Drop-In: /etc/systemd/system/kubelet.service.d            └─10-kubeadm.conf    Active: active (running) since Tue 2020-10-20 05:06:17 UTC; 6min ago      Docs: https://kubernetes.io/docs/home/  Main PID: 6904 (kubelet)     Tasks: 13 (limit: 4678)    CGroup: /system.slice/kubelet.service            └─6904 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --network-plOct 20 05:12:56 node01 kubelet[6904]: E1020 05:12:56.104242	6904 kubelet.go:2183] node "node01" not foundOct 20 05:12:56 node01 kubelet[6904]: I1020 05:12:56.121262	6904 kubelet_node_status.go:70] Attempting to register node node01Oct 20 05:12:56 node01 kubelet[6904]: E1020 05:12:56.122256	6904 kubelet_node_status.go:92] Unable to register node "node01" with API server: Post "https://172.17.0.39:6553/api/v1/nodes"

If we look at the last line of the systemctl status command, we’ll see that the kubelet cannot communicate with the API server.

To get information about the API server, go to the Kubernetes master node and run kubectl cluster-info.

$ kubectl cluster-infoKubernetes master is running at https://172.17.0.39:6443KubeDNS is running at https://172.17.0.39:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

As you can see, there is a mismatch between the port in which the API server is running (6443) and the one to which kubectl is trying to connect (6553). Here we’ll follow the same step as in scenario 2, that is, identify the configuration file.

# cd /etc/systemd/system/kubelet.service.d/$ cat 10-kubeadm.conf# Note: This dropin only works with kubeadm and kubelet v1.11+[Service]Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"

This time, though, you should check /etc/kubernetes/kubelet.conf, where the details about ApiServer are stored.

If you open this file, you’ll see that the port number defined for ApiServer is wrong; it should be 6443, as we saw in the output of kubectl cluster-info.

# cat /etc/kubernetes/kubelet.confapiVersion: v1clusters:- cluster:    certificate-authority-data:    server: https://172.17.0.39:6553  name: default-cluster

Change port 6553 to 6443 and restart the kubelet daemon.

# systemctl daemon-reload# systemctl restart kubelet

Debugging Control Plane

The Kubernetes control plane is the brains behind Kubernetes and is responsible for managing the Kubernetes cluster. To debug control plane failure, we need to follow the same systematic approach and tools we used while debugging worker node failures. In this case, we will try to replicate the scenario where we are trying to deploy an application and it is failing.

Pod in a Pending State

What are some of the reasons that a pod goes into a Pending state?

  • The cluster doesn’t have enough resources.
  • The current namespace has a resource quota.
  • The pod is bound to a persistent volume claim.
  • You’re trying to deploy an application, and it’s failing.
Commands to Run to Identify the Issue

We will start our debugging with kubectl get all commands. So far, we have used kubectl get pods and nodes, but to list all the resources, we need to pass all command line parameters. The command looks like this:

$ kubectl get allNAME                        READY     STATUS      RESTARTS    AGEpod/app-586bddbc54-xd8hs    0/1       Pending     0           16sNAME                  TYPE          CLUSTER-IP    EXTERNAL-IP     PORT(S)     AGEservice/kubernetes    ClusterIP     10.96.0.1     <none>          443/TCP     83sNAME                    READY     UP-TO-DATE    AVAILABLE     AGEdeployment.apps/app     0/1       1             0             16sNAME                              DESIRED     CURRENT     READY     AGEreplicaset.apps/app-586bddbc54    1           1           0         16s

As you can see, the pod is in a Pending state. The scheduler is the component responsible for scheduling the pod on the container. Let’s check the scheduler status in the kube-system namespace.

$ kubectl get pods -n kube-systemNAME                                    READY     STATUS              RESTARTS    AGEcoredns-f9fd979d6-48wdf                 1/1       Running             0           5m54scoredns-f9fd979d6-rl55d                 1/1       Running             0           5m54setcd-controlplane                       1/1       Running             0           6m2skube-apiserver-controlplane             1/1       Running             0           6m2skube-controller-manager-controlplane    1/1       Running             0           6m2skube-flannel-ds-amd64-qbn7x             1/1       Running             0           5m44skube-flannel-ds-amd64-wzcmn             1/1       Running             0           5m53skube-proxy-b645c                        1/1       Running             0           5m53skube-proxy-m4lnk                        1/1       Running             0           5m44skube-scheduler-controlplane             0/1       CrashLoopBackOff    5           5m2s

As you can see in the output, the kube-scheduler-controlplane pod is in CrashLoopBackOff state, and we already know that when the pod is in this state, it will try to start but will crash.

The next command we’ll run is the kubectl describe command to get detailed information about the pod.

$ kubectl describe pod kube-scheduler-controlplane -n kube-systemEvents:  Type      Reason      Age                     From                    Message----      ------      ----                    ----                    -------  Normal    Pulled      5m16s (x5 over 6m44s)   kubelet, controlplane   Container image "k8s.gcr.io/kube-scheduler:v1.19.0" already present on machine  Normal    Created     5m16s (x5 over 6m44s)   kubelet, controlplane   Created container kube-scheduler  Warning   Failed      5m16s (x5 over 6m44s)   kubelet, controlplane   Error: failed to start container "kube-scheduler": Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "exec: \"kube-schedulerror\": executable file not found in $PATH": unknown  Warning   BackOff     103s (x27 over 6m42s)   kubelet, controlplane   Back-off restarting failed container

As you can see, the name of the scheduler is incorrect. To fix this, go to the Kubernetes manifests directory and open kube-scheduler.yaml.

# cd /etc/kubernetes/manifests/$ cat kube-scheduler.yamlapiVersion: v1kind: Podmetadata:  creationTimestamp: null  labels:    component: kube-scheduler    tier: control-plane  name: kube-scheduler  namespace: kube-systemspec:  containers:  - command:    - kube-schedulerror    - --authentication-kubeconfig=/etc/kubernetes/scheduler.conf    - --authorization-kubeconfig=/etc/kubernetes/scheduler.conf    - --bind-address=127.0.0.1    - --kubeconfig=/etc/kubernetes/scheduler.conf    - --leader-elect=true    - --port=0

Under the command section, the name of the scheduler is wrong. Change it from kube-schedulerror to kube-schedule.

Once you fix it, you’ll see that it will recreate the pod, and the application pod will schedule a worker node.

$ kubectl get podsNAME                    READY   STATUS                RESTARTS    AGEapp-586bddbc54-xd8hs    1/1     Running               0           13m$ kubectl get allNAME                        READY     STATUS      RESTARTS    AGEpod/app-586bddbc54-xd8hs    1/1       Running     0           14mNAME                  TYPE          CLUSTER-IP      EXTERNAL-IP     PORT(S)     AGEservice/kubernetes    ClusterIP     10.96.0.1       <none>          443/TCP     15mNAME                    READY     UP-TO-DATE    AVAILABLE     AGEdeployment.apps/app     1/1       1             1             14mNAME                              DESIRED     CURRENT     READY     AGEreplicaset.apps/app-586bddbc54    1           1           1         14mNote: This is the other useful command you can use to debug the issue:$ kubectl logs kube-scheduler-controlplane -n kube-system --tail=10 # (to get the last 10 lines)

Summary

In this article, we have covered some of the ways to debug the Kubernetes cluster. Because Kubernetes is dynamic, it's hard to cover all use cases, but some of the techniques we’ve discussed will help you get started on your Kubernetes debugging journey.

If you want to go beyond debugging Kubernetes and debug the application itself, Sidekick provides a rich feature set with production debugging with tracepoints.

You can check Sidekick's documentation here, and if you still haven't stepped into Sidekick yet, start your journey here.

Read More

Live Debugging
How to Test in Production with Live Debugging
How do you test a system built with third-party services? You can’t replicate the entire architecture on your local machine, so what’s the alternative?
Yasin Kalafat
3 mins read
Live Debugging
Production Debuggers - 2022 Benchmark Results - Part3
This third case is about comparing overheads of Sidekick & Rookout & agentless with active tracepoints under high load
Baris Kaya
3 mins read
Live Debugging
Production Debuggers - 2022 Benchmark Results - Part2
In part2 of this research, we compare Sidekick, Lightrun, and Rookout with active tracepoints under technical limitations
Baris Kaya
4 mins read

Fresh insights from Sidekick experts every week.

Get the best insights, delivered straight to your inbox.
Docs
Security
Integrations
Contact Us
Resources
Blog
Sidekick vs. Rookout
Pricing
Sidekick vs. Lightrun
1 Marina Park Drive STE 1100, Boston, MA
© 2022 Thundra, Inc. All Rights Reserved. Provided by Thundra.
Terms of ServicePrivacy Policy