Use the Conformity Knowledge Base AI to help improve your Cloud Posture

Enable Envelope Encryption for EKS Kubernetes Secrets

Trend Cloud One™ – Conformity is a continuous assurance tool that provides peace of mind for your cloud infrastructure, delivering over 1000 automated best practice checks.

Risk Level: High (not acceptable risk)

Use AWS Key Management Service (KMS) keys to provide envelope encryption of Kubernetes secrets stored in Amazon Elastic Kubernetes Service (EKS), in order to meet security and compliance requirements. Implementing envelope encryption of Kubernetes secrets is considered a security best practice for applications that store sensitive and confidential data. Set up your own AWS KMS Customer Master Key (CMK) and associate the key with your Amazon EKS cluster. When secrets are stored using the Kubernetes secrets API, they are encrypted with a Kubernetes-generated data encryption key, which is then further encrypted using the associated KMS CMK that you have created.

Security

When working with security-critical data, it is strongly recommended to enable encryption of Kubernetes secrets in order to protect your data from unauthorized access and fulfill compliance requirements for data-at-rest encryption within your organization. For example, a compliance requirement is to protect sensitive data that could potentially identify a specific individual such as Personally Identifiable Information (PII), usually used for financial processing systems and healthcare services.


Audit

To determine if your Kubernetes secrets are encrypted with Customer Master Keys using envelope encryption, perform the following actions:

Using AWS Console

01 Sign in to AWS Management Console.

02 Navigate to Amazon EKS console at https://console.aws.amazon.com/eks/.

03 In the left navigation panel, under Amazon EKS, select Clusters.

04 Click on the name (link) of the EKS cluster that you want to examine to access the cluster configuration settings.

05 Select the Details tab and check the Secrets encryption configuration attribute status available in the Secrets encryption section. If the attribute status is set to Disabled, the envelope encryption of the Kubernetes secrets with KMS Customer Master Keys (CMKs) is not enabled for the selected cluster.

06 Repeat step no. 4 and 5 to determine the Kubernetes secrets encryption configuration status for other AWS EKS clusters available within the current region.

07 Change the AWS region from the navigation bar and repeat the process for other regions.

Using AWS CLI

01 Run list-clusters command (OSX/Linux/UNIX) using custom query filters to list the names of all AWS EKS clusters available in the selected region:

aws eks list-clusters
	--region us-east-1
	--output table
	--query 'clusters'

02 The command output should return a table with the requested EKS cluster identifiers:

-------------------------
|     ListClusters      |
+-----------------------+
|  cc-eks-prod-cluster  |
|  cc-eks-main-cluster  |
+-----------------------+

03 Run describe-cluster command (OSX/Linux/UNIX) using the name of the Amazon EKS cluster that you want to examine as identifier parameter and custom query filters to describe the Amazon Resource Name (ARN) of the Customer Master Key used to encrypt the Kubernetes secrets created for the selected cluster:

aws eks describe-cluster
	--region us-east-1
	--name cc-eks-prod-cluster
	--query 'cluster.encryptionConfig[*].provider.keyArn'

04 The command output should return the requested resource ARN:

null

If describe-cluster command output returns null, as shown in the example above, the envelope encryption of the Kubernetes secrets with KMS Customer Master Keys (CMKs) is not enabled for the selected Amazon EKS cluster.

05 Repeat step no. 3 and 4 to determine the Kubernetes secrets encryption configuration status for other AWS EKS clusters deployed in the selected region.

06 Change the AWS region by updating the --region command parameter value and repeat steps no. 1 – 5 to perform the audit process for other regions.

Remediation / Resolution

To enable envelope encryption of Kubernetes secrets using KMS Customer Master Keys (CMKs) for your existing Amazon EKS clusters, you have to re-create your clusters with the required encryption configuration, by performing the following actions:

Using AWS Console

01 Sign in to AWS Management Console.

02 Navigate to KMS console at https://console.aws.amazon.com/kms/.

03 In the left navigation panel, click Customer managed keys.

04 Select the appropriate AWS region from the navigation bar (must match the region where your Amazon EMR cluster is running).

05 Click Create Key button from the dashboard top menu to initiate the setup process.

06 For Step 1 Configure key, choose Symmetric from the Key type section, and select KMS for the Key material origin, available under Advanced options. Click Next to continue.

07 For Step 2 Add labels, provide a unique name (alias) and a short description for your new KMS CMK, then use the Add tag button to create any required tag sets (optional). Click Next to continue the setup process.

08 For Step 3 Define key administrative permissions, choose which IAM users and/or roles can administer your new CMK through the KMS API. You may need to add additional permissions for the users or roles to administer the key from the AWS console. Click Next to continue.

09 For Step 4 Define key usage permissions, within This account section, select which IAM users and/or roles can use the new Customer Master Key (CMK) for cryptographic operations. (Optional) In the Other AWS accounts section, click Add another AWS account and enter an external account ID in order to specify another AWS account that can use this CMK to encrypt and decrypt your EMR log files. The owners of the external AWS accounts must also provide access to this CMK by creating appropriate policies for their IAM users. Click Next to continue the process.

10 For Step 5 Review and edit key policy, review the key policy, then click Finish to create your new KMS Customer Master Key (CMK). Once the key is successfully created, the KMS console will display the following confirmation message: "Success. Your customer master key was created with alias <key-alias> and key ID <key-id>".

11 Navigate to Amazon EKS console at https://console.aws.amazon.com/eks/.

12 In the left navigation panel, under Amazon EKS, select Clusters.

13 Click on the name (link) of the EKS cluster that you want to re-create and collect all the configuration information available for the selected resource.

14 Go back to the Clusters page and choose Create cluster to initiate the new EKS cluster setup process.

15 On the Configure cluster page, perform the following operations:

  1. Provide a unique name for the new EKS cluster in the Name box.
  2. From the Kubernetes version dropdown list, choose the appropriate Kubernetes version.
  3. From the Cluster Service Role dropdown list, select the IAM Role to allow the Kubernetes control plane to manage the required AWS resources on your behalf.
  4. Switch on Enable envelope encryption of Kubernetes secrets using KMS button and select the name (alias) of the newly created Customer Master Key (CMK) from the KMS Key dropdown list.
  5. Click Next to continue the setup process.

16 On the Specify networking page, configure the network settings for the new cluster based on the configuration information taken from source cluster at step no. 13. Click Next to continue.

17 On the Configure logging page, enable the required diagnostic logs, then click Next to continue the process.

18 On the Review and create page, review the cluster configuration details, then click Create to launch your new Amazon EKS cluster.

19 To avoid any unexpected charges on your AWS cloud bill, you can shut down the source cluster. To terminate the source EKS cluster, perform the following actions:

20 Go back to the navigation panel, and under Amazon EKS, choose Clusters.

21 Select the source EKS cluster that you want to shut down.

22 Click on the Delete button from the dashboard top menu to initiate the removal process.

23 In the Delete cluster: <cluster-name> confirmation box, type the cluster name in the required field, then click Delete to terminate the resource.

24 Repeat steps no. 13 – 23 to enable encryption of Kubernetes secrets for other Amazon EKS clusters available within the current region.

25 Change the AWS region from the navigation bar to repeat the entire process for the other regions.

Using AWS CLI

01 Define the policy that enables the selected IAM users and/or roles to manage the new Customer Master Key (CMK), and to encrypt/decrypt your Kubernetes secrets using the AWS KMS API. Create a new policy document (JSON format), name the file eks-cmk-policy.json, and paste the following content (replace the highlighted details, i.e. the ARNs for the IAM users and/or roles, with your own environment details):

{
    "Id": "eks-k8-secrets-cmk-policy",
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Enable IAM User Permissions",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::123456789012:root"
            },
            "Action": "kms:*",
            "Resource": "*"
        },
        {
            "Sid": "Allow access for Key Administrators",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::123456789012:role/AmazonEKSManager"
            },
            "Action": [
                "kms:Create*",
                "kms:Describe*",
                "kms:Enable*",
                "kms:List*",
                "kms:Put*",
                "kms:Update*",
                "kms:Revoke*",
                "kms:Disable*",
                "kms:Get*",
                "kms:Delete*",
                "kms:TagResource",
                "kms:UntagResource",
                "kms:ScheduleKeyDeletion",
                "kms:CancelKeyDeletion"
            ],
            "Resource": "*"
        },
        {
            "Sid": "Allow use of the key",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::123456789012:role/AmazonEKSAdmin"
            },
            "Action": [
                "kms:Encrypt",
                "kms:Decrypt",
                "kms:ReEncrypt*",
                "kms:GenerateDataKey*",
                "kms:DescribeKey"
            ],
            "Resource": "*"
        },
        {
            "Sid": "Allow attachment of persistent resources",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::123456789012:role/AmazonEKSAdmin"
            },
            "Action": [
                "kms:CreateGrant",
                "kms:ListGrants",
                "kms:RevokeGrant"
            ],
            "Resource": "*",
            "Condition": {
                "Bool": {
                    "kms:GrantIsForAWSResource": "true"
                }
            }
        }
    ]
}

02 Run create-key command (OSX/Linux/UNIX) using the policy document created at the previous step (i.e. eks-cmk-policy.json) as value for the --policy parameter, to create your new AWS KMS Customer Master Key (CMK):

aws kms create-key
	--region us-east-1
	--description 'AWS KMS CMK for encrypting EKS Kubernetes secrets'
	--policy file://eks-cmk-policy.json
	--query 'KeyMetadata.Arn'

03 The command output should return the ARN of the new KMS Customer Master Key:

"arn:aws:kms:us-east-1:123456789012:key/abcdabcd-1234-abcd-1234-abcd1234abcd"

04 Run create-alias command (OSX/Linux/UNIX) using the key ARN returned at the previous step to attach an alias to the new CMK. The alias must start with the prefix "alias/" (the command does not produce an output):

aws kms create-alias
	--region us-east-1
	--alias-name alias/KubernetesSecretsCMK
	--target-key-id arn:aws:kms:us-east-1:123456789012:key/abcdabcd-1234-abcd-1234-abcd1234abcd

05 Run describe-cluster command (OSX/Linux/UNIX) using the ID of the Amazon EKS cluster that you want to re-create (see Audit section part II to identify the right resource) to get all the configuration details from the source EKS cluster:

aws eks describe-cluster
	--region us-east-1
	--name cc-eks-prod-cluster
	--query 'cluster'

06 The command output should return the configuration information available for the selected AWS EKS resource:

{
    "status": "CREATING",
    "name": "cc-eks-prod-cluster",
    "roleArn": "arn:aws:iam::123456789012:role/cc-eks-role",
    "resourcesVpcConfig": {
        "vpcId": "vpc-abcdabcd",
        "subnetIds": [
            "subnet-1234abcd",
            "subnet-abcd1234"
        ],

        ...

        "publicAccessCidrs": [
            "0.0.0.0/0"
        ],
        "endpointPublicAccess": true,
        "endpointPrivateAccess": false
    },
    "version": "1.17",
    "platformVersion": "eks.2",
    "createdAt": 1597305567.010
}

07 Run create-cluster command (OSX/Linux/UNIX) using the cluster configuration details returned at the previous step as values for the required parameters, to re-create the specified Amazon EKS cluster in order to enable envelope encryption of Kubernetes secrets using the newly created Customer Master Key (CMK), by setting the CMK ARN as value for the --encryption-config provider parameter:

aws eks create-cluster
	--region us-east-1
	--name cc-new-prod-cluster
	--role-arn arn:aws:iam::123456789012:role/cc-eks-role
	--resources-vpc-config subnetIds=subnet-1234abcd,subnet-abcd1234
	--encryption-config resources=secrets,provider={keyArn=arn:aws:kms:us-east-1:123456789012:key/abcdabcd-1234-abcd-1234-abcd1234abcd}
	--query 'cluster.encryptionConfig'

08 The command output should return the encryption configuration metadata available for the new EKS cluster:

[
    {
        "resources": [
            "secrets"
        ],
        "provider": {
            "keyArn": "arn:aws:kms:us-east-1:123456789012:key/abcdabcd-1234-abcd-1234-abcd1234abcd"
        }
    }
]

09 To avoid any unexpected charges on your AWS bill, you can terminate the source cluster. To shut down the source Amazon EKS cluster, run delete-cluster command (OSX/Linux/UNIX) using the name of the cluster that you want to shut down as identifier parameter:

aws eks delete-cluster
	--region us-east-1
	--name cc-eks-prod-cluster
	--query 'cluster.status'

10 The command output should return the removal request status:

"DELETING"

11 Repeat steps no. 5 – 10 to enable encryption of Kubernetes secrets for other AWS EKS clusters provisioned in the selected region.

12 Change the AWS region by updating the --region command parameter value and repeat steps no. 1 – 11 to perform the entire process for other regions.

References

Publication date Jun 15, 2021