Use the Knowledge Base AI to help improve your Cloud Posture

Private Kubernetes Clusters

Trend Vision One™ provides continuous assurance that gives peace of mind for your cloud infrastructure, delivering over 1100 automated best practice checks.

Risk Level: High (not acceptable risk)

Ensure that your Azure Kubernetes Service (AKS) clusters are deployed as private clusters in order to enhance your Kubernetes workload's security and isolation. Private link-based AKS clusters are Kubernetes clusters where the API server is accessed through a private endpoint within your Virtual Network (VNet), rather than a public IP. This ensures that communication between the API server and the node pools occurs over a secure, private connection, without exposing the cluster to the Internet.

Security

Using private link-based AKS clusters in Azure cloud enhances security by ensuring that the Kubernetes API server is accessible only within your private network, preventing exposure to the public Internet. This reduces the attack surface, improves data privacy, and allows for stricter control over network traffic, aligning with security best practices for sensitive workloads.


Audit

To determine if your Azure Kubernetes Service (AKS) clusters are private, perform the following operations:

Using Azure Portal

01 Sign in to the Microsoft Azure Portal.

02 Navigate to All resources blade at https://portal.azure.com/#browse/all to access all your Microsoft Azure cloud resources.

03 Choose the Azure subscription that you want to access from the Subscription equals all filter box and choose Apply.

04 From the Type equals all filter box, choose Equals, select Kubernetes service, and choose Apply to list only the AKS clusters available in the selected Azure subscription.

05 Click on the name (link) of the AKS cluster that you want to examine.

06 In the resource navigation panel, choose Overview, and select the Properties tab to access the essential configuration properties available for the selected AKS cluster.

07 In the Networking section, check the Private cluster property value. If Private cluster is set to Not enabled, the selected Azure Kubernetes Service (AKS) cluster is not a private link-based AKS cluster.

08 Repeat steps no. 5 – 7 for each AKS cluster provisioned in the selected Azure subscription.

09 Repeat steps no. 3 – 8 for each subscription created in your Microsoft Azure cloud account.

Using Azure CLI

01 Run account list command (Windows/macOS/Linux) with custom output filters to list the IDs of the cloud subscriptions available in your Azure cloud account:

az account list
	--query '[*].id'

02 The command output should return the requested subscription identifiers (IDs):

[
	"abcdabcd-1234-abcd-1234-abcdabcdabcd",
	"abcd1234-abcd-1234-abcd-abcd1234abcd"
]

03 Run account set command (Windows/macOS/Linux) with the ID of the Azure cloud subscription that you want to examine as the identifier parameter to set the selected subscription to be the current active subscription (the command does not produce an output):

az account set
	--subscription abcdabcd-1234-abcd-1234-abcdabcdabcd

04 Run aks list command (Windows/macOS/Linux) with custom query filters to list the name and the associated resource group for each Azure Kubernetes Service (AKS) cluster available in the selected Azure subscription:

az aks list
	--output table
	--query '[*].{name:name, resourceGroup:resourceGroup}'

05 The command output should return the requested AKS cluster names:

Name                    ResourceGroup
----------------------  ------------------------------
cc-project5-aks-cluster  cloud-shell-storage-westeurope
cc-data-mining-cluster   cloud-shell-storage-westeurope

06 Run aks show command (Windows/macOS/Linux) with the name of the AKS cluster that you want to examine (and the associated resource group) as identifier parameters, to describe the Kubernetes API server access profile available for the selected AKS cluster:

az aks show
	--name cc-project5-aks-cluster
	--resource-group cloud-shell-storage-westeurope
	--query '{enablePrivateCluster:apiServerAccessProfile.enablePrivateCluster}'

07 The command output should return the requested configuration information:

{
	"enablePrivateCluster": false
}

If the command output returns null or false for the "enablePrivateCluster" configuration attribute, the selected Azure Kubernetes Service (AKS) cluster is not a private link-based AKS cluster.

08 Repeat steps no. 6 and 7 for each AKS cluster available within the selected Azure subscription.

09 Repeat steps no. 3 – 8 for each subscription created in your Microsoft Azure cloud account.

Remediation / Resolution

A private Azure Kubernetes Service (AKS) cluster uses an internal IP address to ensure that network traffic between the API server and node pools remains on a private network only. To ensure that your AKS cluster is private, you have to re-create your cluster with the required networking configuration. To redeploy your AKS cluster, perform the following operations:

Using Azure Portal

01 Sign in to the Microsoft Azure Portal.

02 Navigate to All resources blade at https://portal.azure.com/#browse/all to access all your Microsoft Azure cloud resources.

03 Choose the Azure subscription that you want to access from the Subscription equals all filter box and choose Apply.

04 From the Type equals all filter box, choose Equals, select Kubernetes service, and choose Apply to list only the AKS clusters available in the selected Azure subscription.

05 Click on the name (link) of the AKS cluster that you want to re-create and copy the necessary configuration details such as the associated resource group, Azure region, Kubernetes version, node size and count, scaling and networking settings and so on. This configuration information will be required later when the new cluster will be deployed.

06 Navigate to Kubernetes Services blade available at https://portal.azure.com/#browse/Microsoft.ContainerService%2FmanagedClusters, choose Create, select Kubernetes cluster, and perform the following actions to deploy your private AKS cluster:

  1. For Basics, provide the following information:
    1. For Subscription, choose your Azure subscription.
    2. For Resource group, select the correct resource group.
    3. Choose the appropriate preset configuration from the Cluster preset configuration dropdown list.
    4. Provide a unique name for the new cluster in the Kubernetes cluster name box.
    5. For Region, select the Azure cloud region where the AKS cluster will be deployed.
    6. For Availability zones, select at least two Availability Zones. Availability Zones (AZs) are used to protect your applications from datacenter failures.
    7. For AKS pricing tier, select the appropriate pricing tier (must match the pricing tier of the source cluster).
    8. For Kubernetes version, select the latest stable version of Kubernetes supported by AKS (recommended).
    9. For Automatic upgrade, choose Enabled with patch (recommended) and set the automatic upgrade scheduler based on your application needs.
    10. Select the desired security channel type from the Node security channel type dropdown list to enable node OS auto-upgrades for the new cluster. You can choose between Unmanaged which applies OS updates automatically through OS built-in patching infrastructure (i.e. OS driven security updates), Security Patch which applies fully-managed OS security patches, either by reimaging nodes or live patching without draining pods, based on necessity, with minimal disruption and maintenance windows honored, or Node Image which updates the nodes with a newly patched VHD containing security fixes and bug fixes on a weekly cadence. For more details, see Security channels for node OS image upgrades. Set the security channel scheduler based on your application needs.
    11. For Authentication and Authorization, choose Microsoft Entra ID authentication with Kubernetes RBAC to enable Microsoft Entra ID for authentication and Kubernetes native RBAC for authorization. For Cluster admin ClusterRoleBinding, click on the Choose Microsoft Entra group (link), and select the group(s) of users that will have administrative access to your AKS cluster.
    12. Choose Next to continue the setup process.
  2. For Node pools, configure the primary and optional node pools for the cluster and choose whether to enable virtual nodes. For Node pool OS disk encryption, set Encryption type to Encryption at-rest with a customer-managed key, and select an existing disk encryption set from the Disk encryption set dropdown list. For more details about node pool disk encryption, see this KB page. Choose Next to continue the setup.
  3. For Networking, perform the following operations:
    1. Under Private access, check the Enable private cluster setting checkbox to make your new AKS cluster private.
    2. For Network configuration, choose the Azure Container Networking Interface (CNI) mode that you want to use for your cluster.
    3. Check the Bring your own Azure virtual network setting checkbox and provide your Virtual Network (VNet) details.
    4. Choose whether to enable Cilium dataplane and network policy or use a network policy such as Calico or Azure. Network policies allow you to define rules for ingress and egress traffic between the pods within your cluster.
    5. Choose Next to continue the setup process.
  4. For Integrations, connect your new AKS cluster with additional Azure cloud services such as Azure Container Registry, Istio, and Azure Policy. Choose Next to continue.
  5. For Monitoring, you can enable and configure Container Insights, Prometheus, and Grafana for more comprehensive data on the overall performance and health of your AKS cluster. You can also configure any required alert rules (use the source cluster alert rules). Choose Next to continue.
  6. For Security, ensure that your cluster subscription is protected by Microsoft Defender for Cloud. Enhance your AKS cluster security by enabling Workload Identity and Image Cleaner features. Choose whether to use Azure Key Vault as a secrets store with your Kubernetes cluster via a CSI volume. Select Next to continue.
  7. For Advanced, provide the name of the necessary infrastructure resource group, and choose Next to continue the setup.
  8. For Tags, create the necessary tag sets (must match the tagging of the source cluster), then choose Review + create to validate the configuration information provided during setup.
  9. For Review + create, review the cluster configuration details, then choose Create to launch your new, private AKS cluster. The provisioning process should take a few minutes to complete.

07 Repeat steps no. 5 and 6 for each AKS cluster that you want to redeploy, available in the selected subscription.

08 Repeat steps no. 3 – 7 for each subscription created in your Microsoft Azure cloud account.

Using Azure CLI

01 Run account list command (Windows/macOS/Linux) with custom output filters to list the IDs of the cloud subscriptions available in your Azure cloud account:

az account list
	--query '[*].id'

02 The command output should return the requested subscription identifiers (IDs):

[
	"abcdabcd-1234-abcd-1234-abcdabcdabcd",
	"abcd1234-abcd-1234-abcd-abcd1234abcd"
]

03 Run account set command (Windows/macOS/Linux) with the ID of the Azure cloud subscription that you want to examine as the identifier parameter to set the selected subscription to be the current active subscription (the command does not produce an output):

az account set
	--subscription abcdabcd-1234-abcd-1234-abcdabcdabcd

04 Run aks show command (Windows/macOS/Linux) with the name of the AKS cluster that you want to re-create (and the associated resource group) as identifier parameters, to describe all the configuration information available for the selected AKS cluster. This configuration information will be required later when the new cluster will be deployed:

az aks show
	--name cc-project5-aks-cluster
	--resource-group cloud-shell-storage-westeurope

05 The command output should return the requested configuration information:

{
	"aadProfile": {
		"adminGroupObjectIDs": null,
		"adminUsers": null,
		"clientAppId": null,
		"enableAzureRbac": true,
		"managed": true,
		"serverAppId": null,
		"tenantId": "abcdabcd-1234-abcd-1234-abcdabcdabcd"
	},

	...

	"supportPlan": "KubernetesOfficial",
	"systemData": null,
	"type": "Microsoft.ContainerService/ManagedClusters",
	"upgradeSettings": null,
	"windowsProfile": {
		"adminPassword": null,
		"adminUsername": "azureuser",
		"enableCsiProxy": true
	}
}

06 Run aks create command (OSX/Linux/UNIX) to launch a new Azure Kubernetes Service (AKS) cluster in the selected Azure subscription. Include the --enable-private-cluster command parameter to make the new AKS cluster private:

az aks create
	--name cc-project5-private-aks-cluster
	--resource-group cloud-shell-storage-westeurope
	--location "westeurope"
	--tier standard
	--node-count 3
	--node-vm-size "Standard_B2s"
	--node-osdisk-size 50
	--kubernetes-version "1.29.8"
	--network-plugin azure
	--vnet-subnet-id "/subscriptions/abcdabcd-1234-abcd-1234-abcdabcdabcd/resourceGroups/cloud-shell-storage-westeurope/providers/Microsoft.Network/virtualNetworks/cc-project5-vnet/subnets/subnet-001"
	--load-balancer-sku standard
	--generate-ssh-keys
	--disable-public-fqdn
	--enable-private-cluster

07 The command output should return the configuration information available for the new AKS cluster:

{
	"aadProfile": null,
	"addonProfiles": {
	"azureKeyvaultSecretsProvider": {
		"config": null,
		"enabled": false,
		"identity": null
	},
	"azurepolicy": {
		"config": null,
		"enabled": false,
		"identity": null
	}
	},
	"apiServerAccessProfile": null,
	"autoUpgradeProfile": {
		"upgradeChannel": "none"
	},

	...

	"storageProfile": {
	"blobCsiDriver": null,
	"diskCsiDriver": {
		"enabled": true
	},
	"fileCsiDriver": {
		"enabled": true
	},
	"snapshotController": {
		"enabled": true
	}
	},
	"supportPlan": "KubernetesOfficial",
	"systemData": null,
	"type": "Microsoft.ContainerService/ManagedClusters",
	"windowsProfile": null,
	"workloadAutoScalerProfile": {
		"keda": null
	}
}

08 Repeat steps no. 6 and 7 for each AKS cluster that you want to re-create, available in the selected subscription.

09 Repeat steps no. 3 – 8 for each subscription created in your Microsoft Azure cloud account.

References

Publication date Oct 22, 2024