All you need to know about moving to containerd on Amazon EKS

The dockershim, an application programming interface (API) shim between the kubelet and the Docker Engine, deprecated from Kubernetes 1.24 in favor of supporting Container Runtime Interface (CRI) compatible runtimes. Amazon Elastic Kubernetes Service (Amazon EKS) also ended support of the dockershim starting with the Kubernetes version 1.24 release. The official EKS Amazon Machine Images(AMI) for version 1.24 and later include containerd as the only runtime, while versions earlier than 1.24 included both the Docker Engine and containerd with Docker Engine as the default runtime.

In this post, we’ll walk you through the context around Dockershim removal and the implications when moving from Docker Engine to containerd runtime to help you prepare run your Kubernetes workloads on it. We’ll also discuss dependency scenarios, along with various AWS native and open-source tools for considerations.

A container runtime is a software component that run containers and manages container images on a node. Docker Engine was the first container runtime leveraged by Kubernetes. Hence, as part of the development, Docker Engine support was inadvertently hardcoded into Kubernetes. With increased Kubernetes adoption, users wanted a choice of container runtimes to cater to different use cases. With multiple container runtimes around, there was a need to provide flexibility that allows smooth interoperability of different container runtimes. Thus, Kubernetes community released Container Runtime Interface (CRI), a plug-in interface to support communication between the kubelet and container runtimes without the need to hard code runtime’s APIs into the Kubernetes code base.

Docker Engine is not Container Runtime Interface(CRI) compatible because it existed before the CRI was created. To solve this, Dockershim was introduced as part of kubelet component to fill in the gap between Docker Engine and CRI (Figure 1).

The fact that kubelet has a tight dependency on specific container runtime led to maintenance burden. To remove this dependency and move toward a collaborative community in support of open standards, the Kubernetes community announced the decision to deprecate Dockershim. Under the hood, Docker Engine was using containerd. Dockershim removal eliminated one extra hop in the stack (Figure 1).

In general, the Kubernetes developer community is in favor of having to develop and maintain only the “Kubernetes core” and abstract out all other supporting components like container runtime, storage, networking, etc., through a mechanism of having standard interface and implementation specific plugins. This de-coupling allows the supporting components having release cycles independent of core Kubernetes releases.

Figure 1: Removal of Dockershim and moving to containerd [Src: https://kubernetes.io/docs/tasks/administer-cluster/migrating-from-dockershim/check-if-dockershim-removal-affects-you/#role-of-dockershim]

Containerd was one component of a decomposition of the original Docker engine into discrete components, including the runc executor and containerd as a lifecycle manager for runc. It was subsequently donated to the Cloud Native Computing Foundation (CNCF) after increasing scope to add image management and registry interactions to stand alone as a core container runtime with an emphasis on simplicity, robustness, and portability, provided as a daemon that can manage the complete container lifecycle. Containerd has a much smaller scope than Docker that results in a smaller codebase that’s easier to maintain and support over time. The Container Runtime Interface (CRI) plugin is built into containerd since version 1.1, which is enabled by default and interacts with containerd through direct function calls. This eliminates another grpc hop in the stack (Figure 2).

Figure 2: CRI plugin built into containerd [Src: https://kubernetes.io/blog/2018/05/24/kubernetes-containerd-integration-goes-ga/]

As described in the introduction, Amazon EKS official AMIs have containerd as the only runtime starting with Kubernetes version 1.24. Amazon EKS Fargate and Bottlerocket already use containerd only.

With containerd, you can still use container images built from “docker build” as they are Open Container Initiative (OCI)-compatible. Any OCI-compliant image, regardless of the tool you use to build it, will look the same to Kubernetes.

Kubernetes:

Amazon EKS:

Containerd:

Impact on your application: Scenarios where there is a dependency with Docker Engine runtime

Some container applications like metrics collector or security agents have the Docker daemon’s socket (/var/run/docker.sock) mounted as a volume. These container images invoke Docker APIs to interact with underlying Docker runtime through Docker socket for collecting metrics or monitoring events, etc. In the absence of Docker runtime, these APIs fail to gather information.

This is because the containerd’s runtime interaction protocol is completely different from Docker Engine’s. The communication with Docker Engine happens over a RESTful HTTP based API, while the communication with containerd happens over CRI protocol which is implemented using gRPC, which is an open-source remote procedure call (RPC) framework.

In this scenario, you must migrate the applications using runtime APIs, from the Docker Engine API to containerd’s gRPC API.

Apart from this, you also need to mount appropriate container runtime socket for communication. For containerd, it is /run/containerd/containerd.sock.

Most of the widely used third-party independent software vendors who package their solutions in containers, employ a mechanism to detect the underlying container runtime and invoke corresponding client-libraries. Another option is to abstract away from the runtime, and rely only on CRI APIs, so that the applications is portable across all CRI compliant runtimes.

The containerd runtime generates logs as a non-JSON string while Docker Engine uses JSON format to write logs by default. Fluentd’s default configuration is designed to only expect JSON strings so when the runtime changes to containerd, it fails to parse any non-JSON log message from containerd and produce a large amount of parse error messages in its container logs. Here is an open issue in Fluentd GitHub repository to check more details.

AWS has launched Fluent Bit as the default log solution for Container insights with significant performance gains and flexible parsers. When you’re upgrading to Amazon EKS version 1.24 with Fluentd pre-configured, we encourage you to consider migrating to Fluent Bit. You must delete Fluentd before installing Fluent Bit to prevent Fluent Bit from processing Fluentd’s parse error log messages and to ensure continuity in logging. Here’s a quick start guide.

Customers also host their Continuous Integration (CI) pipelines on Kubernetes, for triggering container builds. As part of this process, the CI tool executes a task that’s essentially a Kubernetes Job. This Kubernetes Job typically is part of the CI pipeline, which takes up the responsibility of building the application, creating a container image and pushing it to the configured image repository. This job will essentially have docker.sock mounted as a volume and issues docker build instructions from within the container (Job).

Now that Docker runtime is not available, and we won’t be able to use containerd to build images, the above CI pipeline breaks with the following error.

Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

Moreover, building images this way also needs elevated privileges with additional security context to be set or privileged mode to be turned on.

The containerd by design is kept minimal to reduce the footprint and inter-dependencies, and hence doesn’t include the tools to build container images. There are few workarounds to mitigate this issue:

Most of the following tools don’t have a dependency on docker daemon, and hence, don’t require elevated privileges. Some of these tools provide an option to execute the container builds as native Kubernetes Jobs (i.e., elevated securityContext not needed). They also provide simple Command Line Interface (CLI) as well, which use the build process that can be executed from within a container or on a specific build machine.

For identifying which of the pods have Docker Engine dependency, we need to figure out which of the pods are having Docker socket volume mounts. Or we can leverage dds, a kubectl krew plugin to scan through all the workload and detect which of the pods have this dependency (https://github.com/aws-containers/kubectl-detector-for-docker-socket).

Sample YAML with docker.sock mounted:

Run kubectl dds command to identify pods with docker.sock mounted:

Ideally, you should be mounting containerd socket. But many of the existing pod deployment artifacts have Dockershim mount. For the purpose of backward compatibility, the Dockershim volume mounts won’t error out because with Amazon EKS optimized AMIs, there is a soft-link pointing to the current runtime (i.e., containerd). This soft-link is added to enable a smooth migration. If you are building your own custom AMIs, you need to ensure you either create this soft-link or update the mount host path to /run/containerd/containerd.sock.

With this pod, artifacts continue to have a Dockershim socket as part of the volume.

under the hoods, we have:

Current mount points of Dockershim can continue to operate, but underneath containerd socket is being mounted. It is the client program within the container, that will need to identify the runtime configured on the node and use corresponding APIs for interaction.

At times you might want to login to worker nodes, to inspect containers and troubleshoot. Docker CLI debugging capabilities like docker ps/inspect/exec will no longer be an option. Hence, let’s look at a brief summary on containerd based interaction.

You can interact with containerd using various CLIs:

Out of these only ctr is pre-packaged. Other tools will need to be installed manually on need basis.

An important note while working with ctr or in general containerd runtimes. Containerd supports namespaces at the container runtime level. These namespaces are entirely different from the Kubernetes namespaces. Containerd namespaces are used to provide isolation to different applications that might be using containerd like docker, kubelet, etc. One such namespace configured by default for use by kubelet is k8s.io. Specify this namespace, with any of the CLI tools you choose to use.

The primary use-case of ctr is for containerd developers testing daemon or debugging low-level OCI container runtimes since it is close to the actual containerd API. As ctr is hard to use and not user-friendly, nerdctl or crictl are more preferred for general purpose.

If you are used to Docker commands in the past, you can install another tool nerdctl. This is Docker compatible CLI for containerd that provides similar experience as Docker CLI. This isn’t installed on the Amazon EKS nodes out-of-the box, but the binary can be downloaded and run on nodes with containerd runtime. The binary of nerdctl can be downloaded from the releases page. The only point to keep in mind is to specify the namespace of k8s.io while running the commands.

Example:

For more details and samples, check out the GitHub repository at https://github.com/containerd/nerdctl

crictl is a CLI for CRI-compatible container runtimes primarily to inspect and debug container runtimes and images on a Kubernetes node. If you use crictl to create pods on a running Kubernetes cluster, then the kubelet eventually deletes them because it’s not a general purpose tool but for debugging. crictl and its source are hosted in the cri-tools repository. Most of commands are similar to Docker (ex-crictl exec, crictl image, crictl logs) with some unsupported features and only supported features in crictl. You can check mapping from Docker CLI to crictl here (https://kubernetes.io/docs/reference/tools/map-crictl-dockercli/).

In this post, we showed you the context of Dockershim removal, and discussed various considerations and tools validate your Kubernetes workloads when moving from Docker Engine to containerd. We discussed scenarios where you might have a dependency on Docker runtime, and AWS native or open-source alternatives to consider when there is a dependency. Since the Docker CLI will no longer be available, we saw how we can use other tools like ctr, nerdctl, and crictl to interact with the container runtime to troubleshoot containers at node level.

Further reading

Kubernetes is Moving on From Dockershim: Commitments and Next Steps

Amazon EKS ended support for Dockershim

Enable the containerd runtime bootstrap flag

Amazon EKS now supports Kubernetes version 1.24