- Knowledge Base
- Amazon Web Services
- Amazon Managed Streaming for Apache Kafka
- Enable Mutual TLS Authentication for Kafka Clients
Ensure that your Amazon MSK clusters are configured to use mutual TLS (mTLS) authentication in order to allow only trusted clients to connect to your Apache Kafka clusters using Transport Layer Security (TLS) certificates.
Amazon Managed Streaming for Kafka (MSK) is a fully managed AWS cloud service that enables you to migrate, build, and run real-time streaming applications on Apache Kafka. TLS certificate-based authentication caters well for automation use cases such as enabling Amazon EC2 instances to connect to Amazon MSK clusters that work with production and business-critical data.
Audit
To determine if mutual TLS authentication is enabled for your Amazon MSK clusters, perform the following operations:
Using AWS Console
01 Sign in to the AWS Management Console.
02 Navigate to Amazon MSK console at https://console.aws.amazon.com/msk.
03 In the main navigation panel, under MSK Clusters, choose Clusters.
04 Click on the name (link) of the cluster that you want to examine, available in the Cluster namecolumn.
05 Select the Properties tab and check the TLS client authentication through AWS Certificate Manager (ACM) attribute value available in the Security settings section. If TLS client authentication through AWS Certificate Manager (ACM) is set to Not enabled, mutual TLS (mTLS) authentication is not enabled for the selected Amazon MSK cluster.
06 Repeat step no. 4 and 5 for each Amazon Managed Streaming for Kafka (MSK) cluster provisioned within the current AWS region.
07 Change the AWS cloud region from the navigation bar and repeat the Audit process for the other regions.
Using AWS CLI
01 Runlist-clusters command (OSX/Linux/UNIX) using custom query filters to list the Amazon Resource Names (ARNs) of the Amazon MSK clusters available in the selected AWS region:
aws kafka list-clusters --region us-east-1 --query 'ClusterInfoList[*].ClusterArn'
02 The command output should return an array with the requested ARNs:
[ "arn:aws:kafka:us-east-1:123456789012:cluster/cc-kafka-cluster/abcd1234-abcd-1234-abcd-1234abcd1234-ab", "arn:aws:kafka:us-east-1:123456789012:cluster/cc-msk-app-cluster/aabbccdd-1234-aabb-1234-aabbccddaabb-cd" ]
03 Rundescribe-cluster command (OSX/Linux/UNIX) using the ARN of the Amazon MSK cluster that you want to examine as the identifier parameter and custom query filters to determine if mutual TLS authentication is enabled for the selected cluster:
aws kafka describe-cluster --region us-east-1 --cluster-arn arn:aws:kafka:us-east-1:123456789012:cluster/cc-kafka-cluster/abcd1234-abcd-1234-abcd-1234abcd1234-ab --query 'ClusterInfo.ClientAuthentication.Tls.Enabled'
04 The command output should return the mTLS configuration status (true for enabled, false for disabled):
false
If the describe-cluster command output returns false, as shown in the output example above, mutual TLS (mTLS) authentication is not enabled for the selected Amazon MSK cluster.
05 Repeat steps no. 3 and 4 for each Amazon Managed Streaming for Kafka (MSK) cluster available in the selected AWS region.
06 Change the AWS region by updating the --region command parameter value and repeat steps no. 1 – 5 to perform the Audit process for other regions.
Remediation / Resolution
To enable mutual TLS authentication with TLS certificates for your Amazon MSK clusters, perform the following operations:
Note: AWS recommends using independent ACM PCAs for each MSK cluster when you use mutual TLS to control access. Doing so will ensure that TLS certificates signed by PCAs only authenticate with a single MSK cluster.Using AWS CloudFormation
01 CloudFormation template (JSON):
{ "AWSTemplateFormatVersion": "2010-09-09", "Description": "Enable Mutual TLS Authentication for Kafka Clients", "Resources": { "MSKCluster": { "Type": "AWS::MSK::Cluster", "Properties": { "ClusterName": "cc-production-msk-cluster", "KafkaVersion": "3.4.0", "NumberOfBrokerNodes": 2, "BrokerNodeGroupInfo": { "BrokerAZDistribution": "DEFAULT", "ClientSubnets": [ "subnet-0abcd1234abcd1234", "subnet-01234abcd1234abcd" ], "InstanceType": "kafka.m5.large", "SecurityGroups": [ "sg-0abcd1234abcd1234" ], "StorageInfo": { "EbsStorageInfo": { "VolumeSize": 500 } } }, "EncryptionInfo": { "EncryptionInTransit": { "InCluster": true, "ClientBroker": "TLS" } }, "ClientAuthentication": { "Tls": { "Enabled": true, "CertificateAuthorityArnList": [ "arn:aws:acm-pca:us-east-1:123456789012:certificate-authority/abcdabcd-1234-1234-1234-abcdabcdabcd" ] } } } } } }
02 CloudFormation template (YAML):
AWSTemplateFormatVersion: '2010-09-09' Description: Enable Mutual TLS Authentication for Kafka Clients Resources: MSKCluster: Type: AWS::MSK::Cluster Properties: ClusterName: cc-production-msk-cluster KafkaVersion: 3.4.0 NumberOfBrokerNodes: 2 BrokerNodeGroupInfo: BrokerAZDistribution: DEFAULT ClientSubnets: - subnet-0abcd1234abcd1234 - subnet-01234abcd1234abcd InstanceType: kafka.m5.large SecurityGroups: - sg-0abcd1234abcd1234 StorageInfo: EbsStorageInfo: VolumeSize: 500 EncryptionInfo: EncryptionInTransit: InCluster: true ClientBroker: TLS ClientAuthentication: Tls: Enabled: true CertificateAuthorityArnList: - arn:aws:acm-pca:us-east-1:123456789012:certificate-authority/abcdabcd-1234-1234-1234-abcdabcdabcd
Using Terraform (AWS Provider)
01 Terraform configuration file (.tf):
terraform { required_providers { aws = { source = "hashicorp/aws" version = "~> 4.0" } } required_version = ">= 0.14.9" } provider "aws" { profile = "default" region = "us-east-1" } resource "aws_msk_cluster" "msk-cluster" { cluster_name = "cc-production-msk-cluster" kafka_version = "3.4.0" number_of_broker_nodes = 2 broker_node_group_info { instance_type = "kafka.m5.large" client_subnets = ["subnet-0abcd1234abcd1234","subnet-01234abcd1234abcd"] storage_info { ebs_storage_info { volume_size = 500 } } security_groups = ["sg-0abcd1234abcd1234"] } encryption_info { encryption_in_transit { in_cluster = true client_broker = "TLS" } } # Enable Mutual TLS Authentication for Kafka Clients client_authentication { tls { certificate_authority_arns = ["arn:aws:acm-pca:us-east-1:123456789012:certificate-authority/abcdabcd-1234-1234-1234-abcdabcdabcd"] } } }
Using AWS Console
01 Sign in to the AWS Management Console.
02 To use client authentication, you need an ACM Private Certificate Authority (PCA). To create an ACM PCA required for TLS authentication, navigate to AWS Certificate Manager Private Certificate Authority console at https://console.aws.amazon.com/acm-pca/, choose Create a private CA, and perform the following actions:
- For Step 1 Select CA type, select Root CA to create a root Certificate Authority (CA). Choose Next to continue.
- For Step 2 Configure CA subject name, name your CA using the distinguished name (DN) format and provide the required information. Choose Next to continue the setup.
- For Step 3 Configure CA key algorithm, choose RSA 2048 (default)from the Key algorithm options section. Choose Next to continue.
- (Optional) For Step 4 Configure revocation - optional, choose whether to enable certificate revocation list (CRL) distribution or the Online Certificate Status Protocol (OCSP) to communicate revocation status. Choose Next to continue.
- (Optional) For Step 5 Add tags – optional, you can attach tags to your new ACM PCA by using the Add new tag button available in the Tags section. Choose Next to continue.
- (Optional) For Step 6 Configure CA permissions, select the Authorize ACM access to renew certificates requested by this account checkbox to delegate automatic renewal permissions to the ACM service principal. Choose Next to continue.
- For Step 7 Review and create, confirm that your configuration is correct, then choose Confirm and create to create your new ACM Private Certificate Authority (PCA).
- Once your ACM PCA is successfully created, choose Cancel to return to the Private Certificate Authority page.
03 Navigate to Amazon MSK console at https://console.aws.amazon.com/msk.
04 In the main navigation panel, under MSK Clusters, choose Clusters.
05 Click on the name (link) of the cluster that you want to reconfigure.
06 Select the Properties tab and choose Edit from the Security settingssection to modify the access control methods available for the selected cluster.
07 In the Security settings configuration section, select the TLS client authentication through AWS Certificate Manager (ACM) checkbox and choose the ACM PCA created at step no. 2 from the ACM Private CAs dropdown list. Choose Save changes to apply the changes. This will enable mutual TLS authentication for the selected Amazon MSK cluster.
08 Repeat steps no. 2 – 7 for each Amazon MSK cluster that you want to reconfigure, available within the current AWS region.
09 Change the AWS cloud region from the navigation bar and repeat the Remediation process for the other regions.
Using AWS CLI
01 To use client authentication, you need an ACM Private Certificate Authority (PCA). Before you can create an ACM PCA, you must define a CA configuration and save it to a JSON file named ca-config.json. The CA configuration file contains the name and bit size of the private key algorithm, the name of the signing algorithm, and the X.500 certificate subject information:
{ "KeyAlgorithm":"RSA_2048", "SigningAlgorithm":"SHA256WITHRSA", "Subject":{ "Country":"US", "Organization":"TM Corp", "OrganizationalUnit":"IT", "State":"WA", "Locality":"Seattle", "CommonName":"www.domain.com" } }
02 To create an ACM PCA required for mutual TLS authentication, runcreate-certificate-authority command (OSX/Linux/UNIX) using the CA configuration file defined at the previous step (i.e. ca-config.json) as value for the --certificate-authority-configuration parameter:
aws acm-pca create-certificate-authority --region us-east-1 --certificate-authority-configuration file://ca-config.json --certificate-authority-type "ROOT" --idempotency-token 012341234 --tags Key=Name,Value=cc-kafka-pca
03 The command output should return**the ARN of your new ACM PCA:
{ "arn:aws:acm-pca:us-east-1:123456789012:certificate-authority/abcdabcd-1234-1234-1234-abcdabcdabcd" }
04 Runupdate-security command (OSX/Linux/UNIX) using the ARN of the Amazon MSK cluster that you want to reconfigure as the identifier parameter, to update the security settings for the selected cluster in order to enable mutual TLS (mTLS) authentication. Use the ARN of the ACM PCA returned at the previous step as value for the Tls.CertificateAuthorityArnList configuration property:
aws kafka update-security --region us-east-1 --cluster-arn arn:aws:kafka:us-east-1:123456789012:cluster/cc-kafka-cluster/abcd1234-abcd-1234-abcd-1234abcd1234-ab --current-version ABCDABCDABCDAD --client-authentication 'Tls={CertificateAuthorityArnList=["arn:aws:acm-pca:us-east-1:123456789012:certificate-authority/abcdabcd-1234-1234-1234-abcdabcdabcd"],Enabled=true}'
05 The output should return the update-security command request metadata:
{ "ClusterArn": "arn:aws:kafka:us-east-1:123456789012:cluster/cc-kafka-cluster/abcd1234-abcd-1234-abcd-1234abcd1234-ab", "ClusterOperationArn": "arn:aws:kafka:us-east-1:123456789012:cluster/cc-kafka-cluster/abcd1234-abcd-1234-abcd-1234abcd1234-ab/123456789012" }
06 Repeat steps no. 1 – 5 for each Amazon MSK cluster that you want to reconfigure, available in the selected AWS region.
07 Change the AWS cloud region by updating the --region command parameter value and repeat steps no. 1 – 6 to perform the Remediation process for other regions.
References
- AWS Documentation
- Amazon MSK FAQs
- Mutual TLS authentication
- Procedure for creating a CA (console)
- Procedure for creating a CA (CLI)
- Updating security settings of a cluster
- Introducing mutual TLS authentication for Amazon MSK as an event source
- AWS Command Line Interface (CLI) Documentation
- kafka
- list-clusters
- describe-cluster
- update-security
- acm-pca
- create-certificate-authority