SageMaker HyperPod Clusters Encrypted with KMS Customer Managed Keys

Risk Level: High (not acceptable risk)

Ensure that the storage volumes attached to your Amazon SageMaker HyperPod clusters are encrypted with KMS Customer Managed Keys (CMKs) instead of AWS-managed keys in order to have a more granular control over the data-at-rest encryption/decryption process and meet compliance requirements.

Security

SageMaker HyperPod clusters often handle sensitive data, including proprietary algorithms and valuable training datasets. While AWS-managed keys offer a basic level of encryption, they can limit customer control over key management, rotation policies, and access. For organizations with stringent compliance requirements like HIPAA, PCI DSS, or FedRAMP, using KMS Customer Managed Keys (CMKs) is a more robust solution. KMS CMKs give you full control over who can use the encryption keys to access your data. This allows you to implement custom key rotation schedules, maintain audit trails of key usage, and integrate with existing enterprise key management workflows. By default, the Amazon EBS volumes attached to your SageMaker HyperPod cluster are encrypted with AWS-managed keys. Using your own CMKs for your SageMaker HyperPod cluster storage volumes gives you full control. The Amazon KMS service makes it easy to create, rotate, disable, and audit your CMKs.

Audit

To determine the encryption status for the storage volumes attached to your Amazon SageMaker HyperPod clusters, perform the following operations:

Checking SageMaker HyperPod cluster storage volumes for encryption settings using the AWS Management Console is not currently supported.

Using AWS CLI

01 Run sagemaker list-clusters command (OSX/Linux/UNIX) to list the name of each SageMaker HyperPod cluster provisioned in the selected AWS cloud region:

aws sagemaker list-clusters
	--region us-east-1
	--query 'ClusterSummaries[*].ClusterName'

02 The command output should return the requested SageMaker HyperPod cluster names:

[
	"cc-project5-hyperpod-cluster",
	"cc-ai-stack-hyperpod-cluster"
]

03 Run sagemaker describe-cluster command (OSX/Linux/UNIX) with the name of the Amazon SageMaker HyperPod cluster that you want to examine as the identifier parameter and custom output filters to describe the Amazon Resource Name (ARN) of the KMS key used to encrypt data stored on the attached EBS storage volumes:

aws sagemaker describe-cluster
	--cluster-name cc-project5-hyperpod-cluster
	--region us-east-1
	--query 'InstanceGroups[*].InstanceStorageConfigs[].EbsVolumeConfig.KmsKeyId'

04 The command output should return the requested KMS key ARN:

[]

If the sagemaker describe-cluster command output returns an empty array, i.e., [], as shown in the example above, the data on the EBS storage volumes attached to your Amazon SageMaker HyperPod cluster is encrypted using an AWS-managed key (default key provided) instead of using a Customer Managed Key (CMK).

05 Repeat steps no. 3 and 4 for each SageMaker HyperPod cluster provisioned in the selected AWS region.

06 Change the AWS cloud region by updating the --region command parameter value and repeat steps no. 1 – 5 to perform the Audit process for other regions.

Remediation / Resolution

To encrypt the data on the EBS storage volumes attached to your Amazon SageMaker HyperPod cluster using your own KMS Customer Master Key (CMK), perform the following operations:

Enabling CMK-based encryption for SageMaker HyperPod cluster storage volumes using the AWS Management Console is not currently supported.

Using AWS CLI

01 Define the IAM policy that enables the selected users and/or roles to manage your new KMS Customer Managed Key (CMK), and to encrypt/decrypt your SageMaker HyperPod cluster data using the KMS API. Create a new policy document (JSON format), name the file hyperpod-cluster-cmk-policy.json, and paste the following content (replace \<aws-account-id\> and \<role-name\> with your own AWS details):

{
	"Id": "hyperpod-cluster-cmk-policy",
	"Version": "2012-10-17",
	"Statement": [
		{
			"Sid": "Enable IAM User Permissions",
			"Effect": "Allow",
			"Principal": {
				"AWS": "arn:aws:iam::<aws-account-id>:root"
			},
			"Action": "kms:*",
			"Resource": "*"
		},
		{
			"Sid": "Allow access for Key Administrators",
			"Effect": "Allow",
			"Principal": {
				"AWS": "arn:aws:iam::<aws-account-id>:role/service-role/<role-name>"
			},
			"Action": [
				"kms:Create*",
				"kms:Describe*",
				"kms:Enable*",
				"kms:List*",
				"kms:Put*",
				"kms:Update*",
				"kms:Revoke*",
				"kms:Disable*",
				"kms:Get*",
				"kms:Delete*",
				"kms:TagResource",
				"kms:UntagResource",
				"kms:ScheduleKeyDeletion",
				"kms:CancelKeyDeletion",
				"kms:RotateKeyOnDemand"
			],
			"Resource": "*"
		},
		{
			"Sid": "Allow use of the key",
			"Effect": "Allow",
			"Principal": {
				"AWS": "arn:aws:iam::<aws-account-id>:role/service-role/<role-name>"
			},
			"Action": [
				"kms:Encrypt",
				"kms:Decrypt",
				"kms:ReEncrypt*",
				"kms:GenerateDataKey*",
				"kms:DescribeKey"
			],
			"Resource": "*"
		},
		{
			"Sid": "Allow attachment of persistent resources",
			"Effect": "Allow",
			"Principal": {
				"AWS": "arn:aws:iam::<aws-account-id>:role/service-role/<role-name>"
			},
			"Action": [
				"kms:CreateGrant",
				"kms:ListGrants",
				"kms:RevokeGrant"
			],
			"Resource": "*",
			"Condition": {
				"Bool": {
					"kms:GrantIsForAWSResource": "true"
				}
			}
		}
	]
}

02 Run create-key command (OSX/Linux/UNIX) with the policy document created in the previous step (i.e.hyperpod-cluster-cmk-policy.json) as value for the --policy parameter, to create your new Amazon KMS Customer Managed Key (CMK):

aws kms create-key
	--region us-east-1
	--description 'CMK for Encrypting SageMaker HyperPod Cluster Data'
	--policy file://hyperpod-cluster-cmk-policy.json
	--query 'KeyMetadata.Arn'

03 The command output should return the ARN of the new Customer Managed Key (CMK):

"arn:aws:kms:us-east-1:123456789012:key/1234abcd-1234-abcd-1234-abcd1234abcd"

04 Run create-alias command (OSX/Linux/UNIX) to attach an alias to your new Customer Managed Key (CMK). The alias must start with the prefix "alias/" (the command does not produce an output):

aws kms create-alias
	--region us-east-1
	--alias-name alias/HyperpodClusterCMK
	--target-key-id "arn:aws:kms:us-east-1:123456789012:key/1234abcd-1234-abcd-1234-abcd1234abcd"

05 Run sagemaker describe-cluster command (OSX/Linux/UNIX) with the name of the Amazon SageMaker HyperPod cluster that you want to update as the identifier parameter and custom output filters to describe the instance group configuration information available for the selected cluster:

aws sagemaker describe-cluster
	--cluster-name cc-project5-hyperpod-cluster
	--region us-east-1
	--query 'InstanceGroups[]'

06 The command output should return the requested configuration information:

[
	{
		"TargetCount": 1,
		"InstanceGroupName": "cc-project5-hyperpod-ig",
		"InstanceType": "ml.t3.medium",
		"LifeCycleConfig": {
			"SourceS3Uri": "s3://sagemaker-cc-project5-hyperpod-cluster-1234abcd-bucket",
			"OnCreate": "on_create.sh"
		},
		"ExecutionRole": "arn:aws:iam::123456789012:role/sagemaker-cc-project5-hyperpod-cluster-1234abcdExecRole",
		"ThreadsPerCore": 1,
		"InstanceStorageConfigs": [
			{
				"EbsVolumeConfig": {
					"VolumeSizeInGB": 150,
					"RootVolume": false
				}
			}
		],
		"Status": "InService",
		"OverrideVpcConfig": {
			"SecurityGroupIds": [
				"sg-0abcd1234abcd1234"
			],
			"Subnets": [
				"subnet-0abcd1234abcd1234"
			]
		},
		"CurrentImageId": "default",
		"DesiredImageId": "default"
	}
]

07 Run sagemaker update-cluster command (OSX/Linux/UNIX) with the instance group configuration information returned at the previous step to update your Amazon SageMaker HyperPod cluster using a different encryption configuration. To encrypt your SageMaker HyperPod cluster data using a customer-provided KMS key, provide the ARN of the Customer Managed Key (CMK) created earlier in the Remediation process as value for the "VolumeKmsKeyId" parameter property:

aws sagemaker update-cluster
	--cluster-name cc-project5-hyperpod-cluster
	--region us-east-1
	--instance-groups '[{
		"InstanceGroupName": "cc-project5-hyperpod-ig",
		"InstanceCount": 1,
		"InstanceType": "ml.t3.medium",
		"LifeCycleConfig": {
			"SourceS3Uri": "s3://sagemaker-cc-project5-hyperpod-cluster-1234abcd-bucket",
			"OnCreate": "on_create.sh"
		},
		"ExecutionRole": "arn:aws:iam::123456789012:role/sagemaker-cc-project5-hyperpod-cluster-1234abcdExecRole",
		"InstanceStorageConfigs": [
			{
				"EbsVolumeConfig": {
					"VolumeSizeInGB": 150,
					"RootVolume": false,
					"VolumeKmsKeyId": "arn:aws:kms:us-east-1:123456789012:key/1234abcd-1234-abcd-1234-abcd1234abcd"
				}
			}
		]
	}]'

08 The command output should return the information available for the updated SageMaker HyperPod cluster:

{
	"ClusterArn": "arn:aws:sagemaker:us-east-1:123456789012:cluster/abcd1234abcd1234",
	"ClusterName": "cc-project5-hyperpod-cluster",
	"ClusterStatus": "InService",
	"CreationTime": "2025-09-09T10:01:54.119000+00:00",
	"FailureMessage": "",
	"InstanceGroups": [
		{
			"CurrentCount": 0,
			"TargetCount": 1,
			"InstanceGroupName": "cc-project5-hyperpod-ig",
			"InstanceType": "ml.t3.medium",
			"LifeCycleConfig": {
				"SourceS3Uri": "s3://sagemaker-cc-project5-hyperpod-cluster-1234abcd-bucket",
				"OnCreate": "on_create.sh"
			},
			"ExecutionRole": "arn:aws:iam::123456789012:role/sagemaker-cc-project5-hyperpod-cluster-1234abcdExecRole",
			"ThreadsPerCore": 1,
			"InstanceStorageConfigs": [
				{
					"EbsVolumeConfig": {
						"VolumeSizeInGB": 150,
						"RootVolume": false,
						"VolumeKmsKeyId": "arn:aws:kms:us-east-1:123456789012:key/1234abcd-1234-abcd-1234-abcd1234abcd"
					}
				}
			],
			"Status": "InService",
			"OverrideVpcConfig": {
				"SecurityGroupIds": [
					"sg-0abcd1234abcd1234"
				],
				"Subnets": [
					"subnet-01234abcd1234abcd"
				]
			},
			"CurrentImageId": "default",
			"DesiredImageId": "default"
		}
	],
	"RestrictedInstanceGroups": [],
	"VpcConfig": {
		"SecurityGroupIds": [
			"sg-0abcd1234abcd1234"
		],
		"Subnets": [
			"subnet-0abcd1234abcd1234",
			"subnet-01234abcd1234abcd"
		]
	},
	"Orchestrator": {
		"Eks": {
			"ClusterArn": "arn:aws:eks:us-east-1:123456789012:cluster/cc-project5-eks-hyperpod"
		}
	},
	"NodeRecovery": "Automatic",
	"NodeProvisioningMode": "Continuous"
}

09 Repeat steps no. 5 – 8 for each SageMaker HyperPod cluster encrypted with an AWS-managed key, available in the selected AWS region.

10 Change the AWS cloud region by updating the --region command parameter value and repeat the Remediation process for other regions.

References

AWS Command Line Interface (CLI) Documentation
list-clusters
describe-cluster
update-cluster
create-key
create-alias

Publication date Oct 14, 2025

Audit

Using AWS CLI

Remediation / Resolution

Using AWS CLI

References

Related SageMaker rules