How to debug a Docker container

I've learned so many things during the migration of this blog to CoreOS that I didn't know how to write about it. Docker is a different beast than anything I've dealt with in the past and I've had issues with permissions, volumes, fleetctl, container's filesystem, and probably some other things that I can't remember.

Amongst all those issues, the thing that really slowed me down that I wish I knew before was how to debug a container. In the beginning, I was helpless when a build or a running container was failing. It was a game of guessing until I realized how to inspect and dig around inside a container.

If you've ever wondered how to SSH into a container, you will find the answer here. But here's the spoiler: you don't need SSH.

So here's what I've discovered.

Dockerfile creates images which runs as containers

I was confused a lot by the difference between images and containers at first. The way I've come to understand this is that you create read-only image with a Dockerfile. Then, you create a read-write container using a copy of the image you built.

An annoying issue I had while debugging containers and images is that I kept having name collisions when starting my containers. There are two ways to think about this. Either you reuse the same container over and over, or you remove your container once it exits and recreate a new one every time.

Reusing the same container has the benefit of being faster to boot up and also have access to the previous data. Recreating a new container every time means scrubbing all your previous data.

Let's start by pretending you have a Rails project you'd like to run as a container.

$ docker build --tag project.com . $ docker images REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE project.com latest 4a2fec3b5858 4 days ago 819.5 MB

The image is now ready to be used in a container.

# Destroy your container when it exits $ docker run --name instance.project.com --rm project.com # Reuse the same container $ docker create --name instance.project.com project.com $ docker start instance.project.com

If you don't specify the --rm flag with the first command, you can expect to have this error:

$ docker run --name instance.project.com project.com $ docker run --name instance.project.com project.com FATA[0000] Error response from daemon: Conflict. The name "instance.project.com" is already in use by container 3f2b3fa24a58. You have to delete (or rename) that container to be able to reuse that name. $ docker run rm instance.project.com $ docker run --name instance.project.com project.com $ docker ps CONTAINER ID IMAGE PORTS NAMES 3f2b3fa24a58 project.com:latest>3000/tcp instance.project.com

Debug a Dockerfile

While managing a working container is certainly helpful, I didn't start by having everything working. My Dockerfiles were not always building images and sometimes they were, but starting a container would crash on startup.

And when that happened, I had no idea how to debug it. So, how do you debug a failed Dockerfile?

Every instruction you set in the Dockerfile is going to be built as a separate, temporary image for the other instruction to build itself on top of. Here's an example of a build process taken from docker's website:

A successful build
$ docker build . Uploading context 10240 bytes Step 1 : FROM busybox Pulling repository busybox ---> e9aa60c60128MB/2.284 MB (100%) endpoint: https://cdn-registry-1.docker.io/v1/ Step 2 : RUN ls -lh / ---> Running in 9c9e81692ae9 total 24 drwxr-xr-x 2 root root 4.0K Mar 12 2013 bin drwxr-xr-x 5 root root 4.0K Oct 19 00:19 dev drwxr-xr-x 2 root root 4.0K Oct 19 00:19 etc drwxr-xr-x 2 root root 4.0K Nov 15 23:34 lib lrwxrwxrwx 1 root root 3 Mar 12 2013 lib64 -> lib dr-xr-xr-x 116 root root 0 Nov 15 23:34 proc lrwxrwxrwx 1 root root 3 Mar 12 2013 sbin -> bin dr-xr-xr-x 13 root root 0 Nov 15 23:34 sys drwxr-xr-x 2 root root 4.0K Mar 12 2013 tmp drwxr-xr-x 2 root root 4.0K Nov 15 23:34 usr ---> b35f4035db3f Step 3 : CMD echo Hello world ---> Running in 02071fceb21b ---> f52f38b7823e Successfully built f52f38b7823e Removing intermediate container 9c9e81692ae9 Removing intermediate container 02071fceb21b

Notice the ---> Running in [SOME HEX], this is a valid image that you can use to retry the failed instruction and see what's happening.

So basically I would like to ssh into that image and figure out what went wrong. If you've followed along, you know that I'll have to run a container. The self-destructing container is really helpful here because I want to inspect the current status of the image. Every modification I make there is irrelevant as the fix will need to be applied to the Dockerfile anyway to build a working image.

$ docker run --rm -it 9c9e81692ae9 /bin/bash

The hash you want to use is the latest intermediate image you can find in the process of building your image. Unless you've specified a USER directive, you should be logged in as root.

Obviously, debugging is a process of analyzing what's going on and it's different for every situations, but usually the way I start debugging is by trying to manually make the instruction that fail work manually and understand the error. When I get the instruction to work, I usually exit the container, update my Dockerfile and repeat the process until I have something working.

Debug a running container

Now, what happens when your Dockerfile builds a correct image and the container runs as expected but for some reason, the networking aspect of your application doesn't work? This is where debugging a running container becomes useful.

When I migrated to CoreOS, my browser couldn't reach my blog. For some reason, I had issue with nginx sending the request upstream to my instance. I needed to monitor logs of both instance running to try to figure out what was happening.

$ docker run --name instance.project.com project.com $ docker exec -it instance.project.com /bin/bash

I'm back using my own Dockerfile and notice how the exec command uses the container, not the image. I want to connect to the running instance!

If you didn't know how to debug a docker container, I hope I shed some lights on how to do it and gave you the tools to feel confident while building and managing your containers.

Get more ideas like this to your inbox

You will never receive spam, ever.