01 Define the access policy that enables the specified IAM users and/or roles to manage the new KMS Customer Master Key and to encrypt/decrypt MSK cluster data using the AWS KMS API. Create a new policy document, name it kafka-cmk-policy.json, and paste the following content (replace the highlighted details, i.e. the ARNs for the IAM users and/or roles, with your own details):
{
"Version": "2012-10-17",
"Id": "aws-kafka-custom-key-policy",
"Statement": [
{
"Sid": "Enable IAM User Permissions",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::123456789012:root"
},
"Action": "kms:*",
"Resource": "*"
},
{
"Sid": "Grant access to CMK manager",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::123456789012:role/KafkaManager"
},
"Action": [
"kms:Create*",
"kms:Describe*",
"kms:Enable*",
"kms:List*",
"kms:Put*",
"kms:Update*",
"kms:Revoke*",
"kms:Disable*",
"kms:Get*",
"kms:Delete*",
"kms:ScheduleKeyDeletion",
"kms:CancelKeyDeletion"
],
"Resource": "*"
},
{
"Sid": "Allow the use of the CMK",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::123456789012:user/KafkaAdmin"
},
"Action": [
"kms:Encrypt",
"kms:Decrypt",
"kms:ReEncrypt*",
"kms:GenerateDataKey*",
"kms:DescribeKey"
],
"Resource": "*"
},
{
"Sid": "Allow attachment of persistent resources",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::123456789012:user/KafkaAdmin"
},
"Action": [
"kms:CreateGrant",
"kms:ListGrants",
"kms:RevokeGrant"
],
"Resource": "*",
"Condition": {
"Bool": {
"kms:GrantIsForAWSResource": "true"
}
}
}
]
}
02 Run create-key command (OSX/Linux/UNIX) using the file name of the policy document created at the previous step (i.e. kafka-cmk-policy.json) as command parameter to create the new AWS KMS CMK:
aws kms create-key
--region us-east-1
--description 'KMS CMK for encrypting AWS MKS data.'
--policy file://kafka-cmk-policy.json
03 The command output should return the new KMS CMK metadata. Copy the key Amazon Resource Name (Arn parameter value - highlighted) as this information will be required later when you need to specify the key required for MSK data encryption:
{
"KeyMetadata": {
"Origin": "AWS_KMS",
"KeyId": "abcdabcd-1234-1234-1234-abcdabcdabcd",
"Description": "KMS CMK for encrypting AWS MSK data.",
"Enabled": true,
"KeyUsage": "ENCRYPT_DECRYPT",
"KeyState": "Enabled",
"CreationDate": 1517237844.370,
"Arn": "arn:aws:kms:us-east-1:123456789012:key/abcdabcd-1234-1234-1234-abcdabcdabcd",
"AWSAccountId": "123456789012"
}
}
04 Run create-alias command (OSX/Linux/UNIX) using the key ARN returned at the previous step to attach an alias to the new CMK. The alias must start with the prefix "alias/" (the command does not return an output):
aws kms create-alias
--region us-east-1
--alias-name alias/KafkaCustomCMK
--target-key-id arn:aws:kms:us-east-1:123456789012:key/abcdabcd-1234-1234-1234-abcdabcdabcd
05 Run describe-cluster command (OSX/Linux/UNIX) using the ARN of the cluster that you want to re-create as identifier (see Audit section to identify the right resources) and custom query filters to get the cluster configuration metadata:
aws kafka describe-cluster
--region us-east-1
--cluster-arn arn:aws:kafka:us-east-1:123456789012:cluster/cc-kafka-cluster/abcd1234-abcd-1234-abcd-1234abcd1234-a
06 The command output should return the requested resource configuration metadata:
{
"ClusterInfo": {,
"BrokerNodeGroupInfo": {
"BrokerAZDistribution": "DEFAULT",
"ClientSubnets": [
"subnet-abcd1234",
"subnet-1234abcd",
"subnet-12341234"
],
"StorageInfo": {
"EbsStorageInfo": {
"VolumeSize": 450
}
},
...
"SecurityGroups": [
"sg-aabbccdd"
],
"InstanceType": "kafka.m5.large"
},
"ClusterName": "cc-kafka-cluster",
"CurrentBrokerSoftwareInfo": {
"KafkaVersion": "2.1.0"
},
"CreationTime": "2019-02-15T10:35:31.353Z",
"NumberOfBrokerNodes": 3,
"EnhancedMonitoring": "DEFAULT"
}
}
07 Define the necessary parameters for the create-cluster command using the configuration metadata returned at the previous step and save the JSON document to a file named msk-cluster-config.json:
{
"BrokerAZDistribution": "DEFAULT",
"ClientSubnets": [
"subnet-abcd1234",
"subnet-1234abcd",
"subnet-12341234"
],
"InstanceType": "kafka.m5.large",
"SecurityGroups": [
"sg-aabbccdd"
],
"StorageInfo": {
"EbsStorageInfo": {
"VolumeSize": 450
}
}
}
08 Run create-cluster command (OSX/Linux/UNIX) to launch your Amazon Managed Streaming for Kafka (MSK) cluster using the configuration parameters defined at the previous step (i.e. msk-cluster-config.json) and the ARN of the AWS KMS Customer Master Key created earlier in the process:
aws kafka create-cluster
--region us-east-1
--cluster-name cc-msk-production-cluster
--kafka-version 2.1.0
--number-of-broker-nodes 3
--broker-node-group-info file://msk-cluster-config.json
--enhanced-monitoring DEFAULT
--encryption-info EncryptionAtRest={DataVolumeKMSKeyId=arn:aws:kms:us-east-1:123456789012:key/abcdabcd-1234-1234-1234-abcdabcdabcd}
09 The command output should return the command request metadata:
{
"ClusterName": "cc-msk-production-cluster",
"State": "CREATING",
"ClusterArn": "arn:aws:kafka:us-east-1:123456789012:cluster/cc-msk-production-cluster/12341234-abcd-abcd-abcd-1234abcd1234-c"
}
10 Repeat step no. 5 – 9 to configure encryption at rest using KMS Customer Master Keys for other Amazon MSK clusters that you need to re-create within the selected region.
11 Change the AWS region by updating the --region command parameter value and repeat steps no. 1 – 10 to perform the entire process for other regions.
Security 