By Magno Logan
For most organizations, Kubernetes has become the de facto container orchestration platform of choice. But as the popular adage goes, with great power comes great responsibility.
It’s no small feat that Kubernetes is used by many companies worldwide. With its use, however, comes the challenge of properly securing it — a result of the steep learning curve and complex environments involved. Two of these main problems are security defaults and misconfiguration issues.
One way to detect these issues early in the development life cycle is by leveraging threat modeling techniques. In the first part of our research, we’ll explore in detail all the aspects and considerations required to properly and thoroughly perform threat modeling in a Kubernetes environment. We’ll take a top-down approach to discuss the higher-level or most important aspects first before drilling down into the intricacies and nuances of the elements of a Kubernetes cluster.
Figure 1. Exposed Kubernetes clusters according to Shodan.io
According to the Threat Modeling Manifesto written by experienced professionals in this field, threat modeling is about analyzing system representations to highlight concerns about security and privacy characteristics. A process to systematically identify, prioritize, and document potential threats, its main goal is to provide system designers and architects with a structured approach to identify threats and vulnerabilities and design appropriate security controls to mitigate them.
But why is it essential to follow a threat model? As aforementioned, threat modeling helps identify and mitigate security issues early (sometimes even before they are created). This allows organizations to save on both the money and time that would have been required to fix these issues. Threat models also help identify complex risks and map data flows for a better understanding of the threats.
It should be noted that in performing threat modeling, there is no one-size-fits-all method. Due to the large variety and complexity of systems and applications, creating a standard or even automating common threat modeling scenarios is hard. This is why it would be beneficial to those who would conduct this process to pay close attention to each step.
According to the Microsoft Security Development Lifecycle (SDL), there are five significant steps involved in the threat modeling process:
These five steps create a systematic approach to identify, mitigate, and validate threats.
Figure 2. The five phases of the threat modeling according to Microsoft SDL
Notably, analyzing the layers of cloud-native systems would help in properly performing these steps.
The security model of cloud-native systems, which Kubernetes is part of, is constructed such that each of the 4Cs, — cloud, cluster, container, and code —builds upon the preceding outermost layer. Therefore, the cybersecurity of the code layer is enhanced by robust security at the base — in the cloud, cluster, and container layers.
It is important to secure all 4 Cs and not just the code level alone, as implementing cybersecurity measures only at the code level cannot compensate for weak security standards in the other underlying layers. Instead, this leaves the entire system vulnerable.
Figure 3. The 4Cs of cloud-native security layers
In this section, we’ll do a deep dive into the cluster layer, covering each component of a Kubernetes cluster and how attackers can take advantage of these if they are not correctly configured. We’ll also discuss the issues regarding the code and container layers, as well as how to secure those.
The architectural design of Kubernetes is grounded on the principle of ephemeral and independent objects that are interlinked. The following diagram highlights the two main components of a Kubernetes cluster: the control plane and the worker nodes. They contain a series of subcomponents, as shown in the illustration.
Figure 4. A diagram of Kubernetes cluster components
The Kubernetes control plane operates as the chief node of the cluster, overseeing the worker nodes. It functions as the central nervous system, maintaining the intricate structure in an operational and optimal state. The following are its main components:
If the control plane is considered as the operation's nerve center, the worker nodes are akin to the system's brawn. They execute and manage all the pods and containers within the cluster. The cluster can contain anywhere from zero to numerous worker nodes, though it is generally not recommended to operate the pods on the same node as the control plane. The following are the main worker node components:
Understanding the ports and protocols utilized by Kubernetes components can be beneficial when operating the system, especially in a setting with stringent network parameters such as a physical data center with network firewalls or virtual networks within a public cloud. The following is a summary of the ports and protocols for the control plane and the worker nodes, respectively.
|etcd server client API
|Self, control plane
|Self, control plane
Figure 5. Kubernetes data flow diagram by CNCF Financial Services User Group
Using this data flow diagram provided by the Cloud Native Computing Foundation (CNCF) Financial Services User Group, we will now analyze some main cluster components and how a threat actor can abuse them.
As mentioned, the Kube API server is the gateway to the Kubernetes API and provides access to both users and service accounts. All communication is encrypted via TLS and provided via port 6443 (443 for managed Kubernetes) by default. The following diagram (recreated from an original illustration featured in Kubernetes.io) shows the main steps that take place for every request to the API server:
Figure 6. Kube API server request steps (recreated from Kubernetes.io)
In our previous articles, we have consistently warned about the dangers of publicly exposed Kubernetes clusters. Still, unless no authentication is enabled or there is a vulnerability in the cluster, there is only so much an attacker can do with unauthenticated access to the kube-apiserver. Here are some API endpoints that accept unauthenticated (anonymous) API requests:
In this case, most companies limit their cluster to being accessible only via specific IP addresses or Classless Inter-Domain Routing (CIDR) ranges, such as a virtual private network (VPN) or the company’s internal network. This reduces the attack surface and limits the blast radius in the event of a compromise.
The kubelet is the agent deployed on every node of the cluster, ensuring that all containers in a pod are operational. It also serves as the agent responsible for implementing node configuration modifications. While it might not be depicted in the principal Kubernetes architecture diagram, even the master node operates a kubelet (and a kube-proxy) agent, permitting the potential execution of additional pods there. However, this practice isn't typically recommended because the containers running Kubernetes and the containers managing the user’s applications should be separated.
By default, port 10250 is utilized by the kubelet API and is open across all nodes within a cluster, encompassing both the API server control plane and worker nodes. Typically, this port is only visible internally and can't be reached by external services. Outside any other authentication method blocking them, requests to the kubelet's API endpoint are considered anonymous by default. One of the undocumented endpoints of the kubelet API is /runningpods, which provides information about all active pods on the node where the kubelet resides. There's also a /run endpoint, which allows users to execute commands directly on the pods.
The security settings of the kubelet hinge on three pivotal elements:
Etcd serves as the primary storage destination for the cluster, housing all cluster objects. Due to its hierarchical and standardized structure, Kubernetes deployments utilize etcd for preserving REST API objects and installation configurations. An exposed etcd can inadvertently lead to significant data leakage. Regrettably, misconfigured etcd instances remain widespread, with over 4,000 of such exposed services detected on Shodan this year.
Figure 7. Exposed etcd instances according to Shodan.io
If an attacker circumvents the API server and directly manipulates objects within etcd, it would equate to unrestricted access to the entire cluster. The intruder could create pods, access secrets, and view sensitive information like user credentials. To mitigate this risk, it's essential to enable encryption in transit and ensure encryption at rest to prevent data leakage or unauthorized alterations to the etcd data. Fortunately, most managed Kubernetes services offer this option or enable it by default.
Figure 8. MITRE ATT&CK for containers
Another approach to the threat modeling for Kubernetes is to leverage the MITRE ATT&CK for containers, the initial version of which was published in 2021 as a result of a collaboration between MITRE and the Trend Micro research team. Currently, it specifies techniques and sub-techniques that threat actors leverage for compromising containers and clusters in the wild. Leveraging this matrix can help organizations focus on the most common weaknesses attackers exploit in these environments. Here we analyze some weaknesses that pertain more to container orchestration and Kubernetes:
Figure 9. Using masscan to search for port 10250 (kubelet API) and access the running pods inside the cluster
Figure 10. The kubectl run command to perform pod or container escape via privileged pod
Figure 11. Attacker leveraging masscan to find exposed worker nodes inside the network
Figure 12. Attackers using zgrab, a banner-grabbing tool, to collect information on running services
Figure 13. Analyzing Kubernetes audit logs on EKS for suspicious pod deployments
Kubernetes has a steep learning curve, and it’s not secured by default. To secure Kubernetes, understanding how the main components work and how attackers can leverage these is essential. Organizations should therefore keep these parts and methods in mind for better security when deploying and managing Kubernetes clusters. For more insights on Kubernetes security, the second part of this article will delve deeper into more threats and vulnerabilities surrounding Kubernetes.
Like it? Add this infographic to your site:
1. Click on the box below. 2. Press Ctrl+A to select all. 3. Press Ctrl+C to copy. 4. Paste the code into your page (Ctrl+V).
Image will appear the same size as you see above.