Mitigating the Threat of Sidecar Container Injection

By Magno Logan

Kubernetes, often abbreviated as K8s, is an open-source container orchestration platform designed to automate application container deployment, scaling, and operation. Containers are lightweight, standalone, executable software packages that include everything needed to run a piece of software, including the code, runtime, system tools, libraries, and settings. Containers are isolated from each other and the host system. Theoretically, they should have only one process running per container; only the main application should be executed inside a container. But there are times when we need to run other processes for things like logging and monitoring, so instead of running both on the same container, we deploy another container; that new container is called a sidecar container.

Sidecar containers in container orchestration

In Kubernetes and container orchestration, a "sidecar" refers to a design pattern in which an additional container runs alongside the main application container in the same pod to augment or enhance its functionality. The sidecar and primary containers share the same lifecycle and local network.

Sidecar containers are handy for modularizing features and functionalities that are not directly related to the main application logic but are still essential for running efficiently. Separating these concerns into different containers helps achieve the single responsibility principle and makes the system more maintainable.

There are a few ways to use sidecar containers for logging purposes legitimately, such as:

• The sidecar container directs application logs to its standard output (Figure 1).
• The sidecar container operates a logging agent to retrieve logs from the application container (Figure 2).

Figure 1. Streaming sidecar container

Figure 2. Sidecar container with a logging agent

In summary, a sidecar container allows you to:

• Operate a supplementary container next to your primary one within the same pod.
• Transfer data from an external source to a local storage accessed by the main container.
• Maintain a common lifecycle, resources, and network environment.

Pros and cons of sidecar containers

These are the main benefits of using sidecar containers:

• Isolation. The core application operates autonomously in a container, while the sidecar manages additional processes and utilities. It conceals the main application's code from external services interacting with it and helps to isolate issues.
• Quick Deployment. Deploying a sidecar container is straightforward, and often simpler than incorporating extra functionalities directly into the primary application container.
• Scalability. After establishing sidecar containers, amplifying their presence to accommodate more pods becomes effortless. Whether you deploy one or a hundred sidecar containers, it remains just as uncomplicated.

On the other hand, the main downsides of using sidecar containers are:

• Resource Utilization. Additional containers lead to increased memory and CPU usage. Often, from a resource standpoint, it's more resource-efficient to consolidate all processes within one container or execute associated tasks at the node level.
• Operational Complexity. Incorporating sidecars boosts the cumulative count of containers to oversee, plus the intricacies of their interrelationships. An extensive monitoring system becomes essential, capable of discerning, for example, the precise impact on main applications due to a sidecar container's failure.
• Synchronization Challenges. Aligning updates between the main application container and its supporting sidecar requires extra effort. Ensuring seamless updates across all intertwined components can be challenging.

Sidecar Injection

The Kubernetes threat matrix aims to provide a structured understanding of the tactics, techniques, and procedures (TTPs) that potential adversaries might employ against Kubernetes setups. Drawing inspiration from MITRE ATT&CK, this matrix offers a snapshot of possible TTPs an attacker might leverage during their assault. The Kubernetes threat matrix also provides specific countermeasures tailored to Kubernetes systems and attack methodologies. One of these techniques under Execution Tactics is called Sidecar Injection (MS-TA9011), which is described as such:

“A Kubernetes Pod is a group of one or more containers with shared storage and network resources. Sidecar container is a term used to describe an additional container that resides alongside the main container. For example, service-mesh proxies operate as sidecars in the applications’ pods. Attackers can run their code and hide their activity by injecting a sidecar container to a legitimate pod instead of running their own separate pod in the cluster.” (Microsoft Threat Matrix for K8s)

This technique is correlated to the MITRE ATT&CK for Containers technique, called Deploy Container (T1610). The Sidecar Injection technique can also be considered when an attacker compromises the current running sidecar containers to modify its behavior. This can be done via the API or the kubectl tool to install new software and execute other binaries, such as cryptocurrency miners.

In a nutshell, sidecar container injection is a technique in which the attacker tries to stay undetected by injecting their malware inside a sidecar container or even deploying a new one, compromising the victim's cluster and staying stealthy at the same time.

Sidecar Injection methods

There are many ways attackers can exploit sidecar injection to compromise containers inside a cluster. Let’s analyze a few of those ways.

Leveraging kubectl

If the attacker is able to install kubectl inside one of the containers, depending on the permissions, the attacker could run this simple script to install a cryptominer on the last container of each pod that has more than one container running. Let’s take this cluster in Figure 3 as an example with these running pods:

Figure 3. List of running pods inside the cluster

Figure 4 shows a simple shell script that could be used to accomplish the task of leveraging kubectl:

Figure 4. Script to execute a shell command in the last container of each pod

This simple script assumes two things:

• kubectl is installed and configured correctly.
• The container has the sh binary, otherwise the command won’t work.

Figure 5 shows the result of doing the above script in Figure 4 on the Azure Kubernetes Service (AKS) cluster:

Figure 5. Output for executing a shell command in the last container of each pod in the AKS cluster

As mentioned earlier, the execution would only work if the sh binary is already installed inside the container, otherwise we will receive the “Executable file not found” error like we got in some of the containers. An improved version could check if sh or other shell binaries are installed, and if not, install them.

Without kubectl

If the attacker is unable to install kubectl to perform sidecar injection, they can still leverage the API Server, curl, and other commands, as long as it has access to the necessary files. The main requirements for this are the Kube API Server URL, the Bearer Token, and the CA certificate file path. Figure 6 shows a sample script with a few more lines than the kubectl one:

Figure 6. Script to execute a shell command in the last container of each pod without kubectl

If the variables are set correctly, this script would generate the same results as the previous one. Again, this script is only focusing on the sh binary, assuming that it is inside the container otherwise it won’t work. The echo command is just to demonstrate that the command has been executed.

Sidecar Injection mitigations

Besides having a proper runtime security solution to monitor all the containers inside the cluster, here are a few other mitigations to reduce the likelihood and impact of these attacks in your Kubernetes environment, according to the Microsoft Threat Matrix for K8s:

• MS-M9003: Adhere to least-privilege principle - Prevent unnecessary users and service accounts from creating new pods and controllers.
• MS-M9013: Restrict over permissive containers - Restrict over permissive containers in the cluster using an admission controller.
• MS-M9005.003: Gate images deployed to Kubernetes cluster - Restrict deployment of new containers from trusted supply chain

Sidecar container best practices

Organizations can protect their Kubernetes setups against this threat to sidecar containers by applying the following security recommendations:

Ensure there's a valid justification for segregating the containers.
Handling sidecar containers becomes intricate, mainly when there are numerous containers. Like any standard container, they require monitoring, updates, and resource allocation. Overlooking these necessities can jeopardize your cluster's efficiency.

Given this intricacy, only incorporate sidecar containers when they offer tangible advantages to your main application. For instance, you could enhance your main application with logging capabilities, or a file system designed in another programming language. A good guideline is that sidecar containers should enhance, not impede, the application's performance.

Aim for concise, modular designs.
Sidecar containers should be designed as streamlined units with restricted capabilities. This simplicity eases maintenance and allows for swift bug detection within the pods. Ensuring the sidecar remains modular and succinct can prevent resource contention with the primary container.

Be mindful of resource boundaries.
Sidecar containers need to avoid overshadowing the main container regarding resource consumption. Sidecars exist primarily to augment or supplement the main container's functions, playing a supporting role. While their compact design can minimize the risk of excessive resource use, setting bounds on their resource access is equally critical. Letting sidecars consume resources unchecked, especially memory, can degrade the performance of your Kubernetes cluster.

One potential issue is an insufficient number of pods to manage workloads. If a node in your cluster exhausts its memory, the system kernel will promptly terminate the most memory-intensive container, leading to an out-of-memory (OOM) error. This usually results in a terminated pod. Properly configuring your sidecar will enable you to set specific resource limits, sidestepping this challenge.

Conclusion

Although a legitimate approach, sidecar containers can add even more complexity to Kubernetes clusters, not only making it harder to manage but also to detect compromises. This article demonstrates that attackers can leverage this technique after compromising a K8s cluster to remain stealthy and avoid detection. Treating your sidecar containers just like any other container in the cluster and monitoring their activities for suspicious or malicious behavior is essential. We recommend creating a new MITRE ATT&CK for the sidecar injection container technique to focus specifically on this method.