The Easiest Way of Debugging Kubernetes Deployments

As more organizations adopt microservices architecture, they are turning to Kubernetes to deploy those applications at scale. Kubernetes is an open-source platform for container orchestration that allows developers to deploy their applications declaratively. You define your application’s state and Kubernetes achieves it.

However, not everything is smooth and easy with Kubernetes. Once the application is deployed to a Kubernetes cluster and you face an error, you are left with logs and Kubernetes resource states to debug your applications. There are no constructs to debug your applications running in the production cluster.

This is where tools like Telepresence, Bridge, and Sidekick come into play. These tools allow you to interactively debug your applications running in Kubernetes. Live debugging allows you to attach a debugger to a running application and get its state, rather than relying on logs and intuition to fix the issue.

In this post, I demonstrate what each of these tools has to offer in terms of functionality and ease of use. But before evaluating these tools, let’s look at the demo application we’ll be using throughout this post.

Demo Setup

For the purpose of this demo, I’ll use two REST APIs, namely backend and frontend. The backend service is written in Node.js and the frontend is written in Python. The subsequent section details the endpoints supported by each service and the input structure for each.

Backend Service

The backend is a node-based REST API with following endpoints.

• Health endpoint: Available at /health. Returns an HTTP 200 if service is running. 
• Calculation endpoint:  Available at /calculate. It takes a JSON input with 3 parameters.
  x: first number
  y: second number
  op: operator

A sample JSON for adding two numbers will look like this.

{
    "x": 10,
    "y": 20,
    "op": "+"
}

Frontend Service

The frontend is a Flask-based API that calls the calculation backend service. This service has the following endpoints. 

• Health endpoint : available at /health and returns a HTTP 200 if service is running.
• Calculation endpoint : available at /calculate. This endpoint takes operation in plain English as input, such as: add X and Y, subtract Y from X, divide X by Y, and multiply X by Y. If there is any other operation, it will throw an error. The sample input is given below. 

{
    "task": "Add 30 and 5"
}

The code for services, Docker images, and Kubernetes resources can be found in the GitHub repository. Once Docker images are built and the applications (both frontend and backend) are deployed on the Kubernetes cluster, you can verify it by accessing the frontend service via cURL. I have exposed the frontend service via a NodePort service, which you can use to connect to it. 

Note that I’ve added the dependencies and code for installing the Sidekick agent that we’ll use in the next section.

Now, let’s assume I’m facing an exception and getting an HTTP 500 status code while trying to send a request to the frontend service. I can refer to the logs to check why I am getting the exception, but since logs do not contain the values of all the variables while the request is executing, it's very difficult to debug what is causing the exception. That’s when remote debugging tools such as Telepresence and Sidekick come in handy. 

In the next section, we’ll see how each of these tools can help you debug your services.

Debugging Live Application

In this section, let’s do a hands-on demo of how to debug an application that’s running in the Kubernetes cluster.

Telepresence

One tool to debug the application is Telepresence, a utility that allows you to access production services running in Kubernetes locally or vice versa. It creates a bidirectional tunnel between your Kubernetes cluster and your local machine.

This allows you to connect local running code to a production service or proxy the service running locally when you request the production service and to debug a Kubernetes service locally. 

Sidekick

Sidekick is an open source tool that allows you to live debug your production applications. The way Sidekick differs from Telepresence is that it doesn’t create a tunnel. You simply have to ship a Sidekick agent with your application. Once you deploy your application, the Sidekick agent will automatically start beside your application and start sending execution metrics to the Sidekick backend. You can read about the process of installing the agent in Java, Node.js and Python in the documentation. I have already added the agent code along with the applications.

You can attach logpoints and tracepoints to your application, and Sidekick will intercept those points and let you view the application state at those points.

Sidekick provides multiple ways to attach and view these logpoints and tracepoints. You can either use Web IDE to attach these points, use REST clients or use plugins for VSCode, WebStorm, PyCharm and IntelliJ and control your agents.

In this section, I’ll do a hands-on demonstration of this. Before getting started, you’ll need to sign up for a free Sidekick account. Once your signup is complete, you’ll get your API key, which you’ll pass to your applications so that the Sidekick agent can connect to the Sidekick backend. 

Next, redeploy your Kubernetes application by passing in the following environment variables so that the Sidekick agent can be initialized.

The next step is to install the VSCode extension for Sidekick. Once installed, you’ll see Sidekick in the left side panel.

You’ll also see the pane for running applications, tracepoints, and logpoints. Next, log in to the plugin by clicking the blue arrow icon.

This will open up the browser where you will be prompted to log in, after which you’ll get the following screen.

After successfully logging in, you’ll see your applications in the left pane.

Here, you can see both services. Now, you can start adding logpoints and tracepoints.

I’ll begin by adding some tracepoints. Select the application you wish to add the tracepoints to, then right click on the line where you wish to add tracepoints and go to Sidekick > Tracepoint.

Once you add the tracepoint, it will show up in the left pane.

Now, every time the production service is called and the code execution reaches this point, the HitCount will increase; and for every hit, the entire state of the stack will be captured.

Let’s call our service once using cURL.

curl -XPOST -H 'Content-Type:application/json' http://:/calculate -d '{"task" : "Add 2 and 4"}'

If everything is set up correctly, the HitCount will increase to 1.

Click on the tracepoint, and another Sidekick pane will appear at the bottom. This pane will have all the hits with a timestamp for when the hit happened.

Click on the tracepoint event, and it will open up all the call frames it has captured and show the values of all the variables captured in that frame. 

Similarly, if you make more requests to your API, the HitCount will keep increasing until the tracepoint’s hit or time limit is reached. I made some more requests and can see the HitCount increasing. 

The trace events can also be seen in the bottom panel. 

By clicking on each individual event, you can trace the state of the application at that point in time.

To add a logpoint, right click on the line to which you wish to add the logpoint, and go to Sidekick > Logpoint.

This will open up the panel on the left to configure the logpoint. You can also use mustache {{}} syntax to log the values of variables.

Once done, hit “Submit,” and that should add a logpoint to your left pane.

Now, when you send requests to your API, the HitCount for logpoint will increase and your log statements will be available in the bottom panel.

In the bottom panel, you can see your log events with the message that you logged. Note that you did not have to add any extra code and redeploy to get the log message. It can simply be added in my IDE, and it worked.

The ability to dynamically add tracepoints and logpoints allows for easy debugging of production applications, without any changes or need to set up services locally. This is just one advantage of Sidekick over a tool like Telepresence, there are a lot more like Exception Stack Collection & living trace points. With Telepresence, you would still need to set up and run your code locally, but with Sidekick you are able to debug the production copy of your application right from within your IDE.

Conclusion

When you have multiple services running in Kubernetes and you wish to debug them, you’re only limited to logs generated by your application. Sometimes just analyzing the logs is not enough to get to the bottom of the error, and you need the ability to debug and see the state of the application at a certain time. That’s when tools like Sidekick are useful. Its ability to add dynamic tracepoints and logpoints without requiring any redeployment opens up a host of possibilities. 

You can add tracepoints on the fly, with each tracepoint capturing the state of variables at that point in time. The logpoints allow you to add logs to specific code lines. You can also capture variable values within the logs. You can achieve all of this without leaving your favorite IDE. Here, I used VSCode to do it, but Sidekick plugins are also available for IntelliJ IDEA, PyCharm & WebStorm.

With Sidekick in your development stack, you can effectively debug your Kubernetes deployments with near zero overhead.