Enable Apache Kafka Latest Security Features

01 Sign in to the AWS Management Console.

02 Navigate to Amazon MSK console at https://console.aws.amazon.com/msk.

03 In the main navigation panel, under MSK Clusters, choose Clusters.

04 Click on the name (link) of the cluster that you want to examine, available in the Cluster name column.

05 In the Cluster summary section, check Apache Kafka version attribute value to determine the Apache Kafka software version used by the selected cluster. If the Apache Kafka version installed on the cluster is lower than 2.5.1, the selected Amazon MSK cluster does not have access to the latest security features introduced by Apache Kafka starting with the version 2.5.1.

06 Repeat step no. 4 and 5 for each Amazon Managed Streaming for Kafka (MSK) cluster provisioned within the current AWS region.

07 Change the AWS cloud region from the navigation bar and repeat the Audit process for the other regions.

01 Runlist-clusters command (OSX/Linux/UNIX) using custom query filters to list the Amazon Resource Names (ARNs) of the Amazon MSK clusters available in the selected AWS region:

aws kafka list-clusters
  --region us-east-1
  --query 'ClusterInfoList[*].ClusterArn'

02 The command output should return an array with the requested cluster ARNs:

[
	"arn:aws:kafka:us-east-1:123456789012:cluster/cc-kafka-cluster/abcd1234-abcd-1234-abcd-1234abcd1234-ab",
	"arn:aws:kafka:us-east-1:123456789012:cluster/cc-msk-app-cluster/aabbccdd-1234-aabb-1234-aabbccddaabb-cd"
]

03 Rundescribe-cluster command (OSX/Linux/UNIX) using the ARN of the Amazon MSK cluster that you want to examine as the identifier parameter and custom query filters to describe the Apache Kafka software version installed on the selected cluster:

aws kafka describe-cluster
  --region us-east-1
  --cluster-arn arn:aws:kafka:us-east-1:123456789012:cluster/cc-kafka-cluster/abcd1234-abcd-1234-abcd-1234abcd1234-ab
  --query 'ClusterInfo.CurrentBrokerSoftwareInfo.KafkaVersion'

04 The command output should return**the Kafka version used by the selected MSK cluster:

"2.3.1"

05 Repeat steps no. 3 and 4 for each Amazon Managed Streaming for Kafka (MSK) cluster available in the selected AWS region.

06 Change the AWS region by updating the --region command parameter value and repeat steps no. 1 – 5 to perform the Audit process for other regions.

01 CloudFormation template (JSON):

{
	"AWSTemplateFormatVersion": "2010-09-09",
	"Description": "Update Apache Kafka Version for MSK Cluster",
	"Resources": {
		"MSKCluster": {
			"Type": "AWS::MSK::Cluster",
			"Properties": {
				"ClusterName": "cc-production-msk-cluster",
				"NumberOfBrokerNodes": 2,
				"EnhancedMonitoring": "DEFAULT",
				"BrokerNodeGroupInfo": {
					"BrokerAZDistribution": "DEFAULT",
					"ClientSubnets": [
						"subnet-0abcd1234abcd1234",
						"subnet-01234abcd1234abcd"
					],
					"InstanceType": "kafka.m5.large",
					"SecurityGroups": [
						"sg-0abcd1234abcd1234"
					],
					"StorageInfo": {
						"EbsStorageInfo": {
							"VolumeSize": 500
						}
					}
				},
				"EncryptionInfo": {
					"EncryptionInTransit": {
						"ClientBroker": "TLS",
						"InCluster": true
					},
					"EncryptionAtRest": {
						"DataVolumeKMSKeyId": "arn:aws:kms:us-east-1:123456789012:key/1234abcd-1234-abcd-1234-abcd1234abcd"
					}
				},
				"KafkaVersion": "3.5.1"
			}
		}
	}
}

02 CloudFormation template (YAML):

AWSTemplateFormatVersion: '2010-09-09'
	Description: Update Apache Kafka Version for MSK Cluster
	Resources:
	MSKCluster:
		Type: AWS::MSK::Cluster
		Properties:
		ClusterName: cc-production-msk-cluster
		NumberOfBrokerNodes: 2
		EnhancedMonitoring: DEFAULT
		BrokerNodeGroupInfo:
			BrokerAZDistribution: DEFAULT
			ClientSubnets:
			- subnet-0abcd1234abcd1234
			- subnet-01234abcd1234abcd
			InstanceType: kafka.m5.large
			SecurityGroups:
			- sg-0abcd1234abcd1234
			StorageInfo:
			EbsStorageInfo:
				VolumeSize: 500
		EncryptionInfo:
			EncryptionInTransit:
			ClientBroker: TLS
			InCluster: true
			EncryptionAtRest:
			DataVolumeKMSKeyId: arn:aws:kms:us-east-1:123456789012:key/1234abcd-1234-abcd-1234-abcd1234abcd
		KafkaVersion: 3.5.1

01 Terraform configuration file (.tf):

terraform {
	required_providers {
		aws = {
			source  = "hashicorp/aws"
			version = "~> 4.0"
		}
	}

	required_version = ">= 0.14.9"
}

provider "aws" {
	profile = "default"
	region  = "us-east-1"
}

resource "aws_msk_cluster" "msk-cluster" {
	cluster_name           = "cc-production-msk-cluster"
	number_of_broker_nodes = 2
	enhanced_monitoring    = "DEFAULT"

	broker_node_group_info {
		instance_type  = "kafka.m5.large"
		client_subnets = ["subnet-0abcd1234abcd1234","subnet-01234abcd1234abcd"]
		storage_info {
			ebs_storage_info {
			volume_size = 500
			}
		}
		security_groups = ["sg-0abcd1234abcd1234"]
	}

	encryption_info {
		encryption_in_transit {
			client_broker = "TLS"
			in_cluster    = true
		}
		encryption_at_rest_kms_key_arn = "arn:aws:kms:us-east-1:123456789012:key/1234abcd-1234-abcd-1234-abcd1234abcd"
	}

	# Update Apache Kafka Version for MSK Cluster
	kafka_version = "3.5.1"
}

01 Sign in to the AWS Management Console.

02 Navigate to Amazon MSK console at https://console.aws.amazon.com/msk.

03 In the main navigation panel, under MSK Clusters, choose Clusters.

04 Click on the name (link) of the cluster that you want to upgrade.

05 Select the Properties tab and choose Upgrade from the Apache Kafka version section to upgrade the Apache Kafka software version installed on the cluster to the latest supported version.

06 On the Upgrade the Apache Kafka version for <cluster-name> page, select the latest Kafka version available for upgrade from the Apache Kafka version dropdown list, choose whether or not to update the cluster configuration together with the upgrade, then select Upgrade to install the latest version of Apache Kafka on the selected Amazon MSK cluster. A progress bar should be displayed in the console during the upgrade process.

07 If required, when you update the Apache Kafka version of an MSK cluster, also check your client-side software to make sure its version enables you to use the features of the cluster's new Apache Kafka version.

08 Repeat steps no. 4 – 6 for each Amazon MSK cluster that you want to update, available within the current AWS region.

09 Change the AWS cloud region from the navigation bar and repeat the Remediation process for the other regions.

01 Runget-compatible-kafka-versions command (OSX/Linux/UNIX) to get the list of the Apache Kafka versions to which you can update your Amazon MSK cluster:

aws kafka get-compatible-kafka-versions
  --cluster-arn arn:aws:kafka:us-east-1:123456789012:cluster/cc-kafka-cluster/abcd1234-abcd-1234-abcd-1234abcd1234-ab
  --query 'CompatibleKafkaVersions'

02 The command output should return the compatible target Apache Kafka versions and the current (source) version installed on the cluster:

[
	{
		"SourceVersion": "2.3.1",
		"TargetVersions": [
			"2.4.1.1",
			"2.6.2",
			"2.7.1",
			"2.8.0",
			"2.6.3",
			"2.7.2",
			"2.8.1",
			"2.5.1",
			"2.6.0",
			"2.6.1",
			"2.7.0"
		]
	}
]

03 Runupdate-cluster-kafka-version command (OSX/Linux/UNIX) using the ARN of the Amazon MSK cluster that you want to update as the identifier parameter, to upgrade the Apache Kafka version for the selected cluster to the latest supported version (i.e. latest target version, returned at the previous step):

aws kafka update-cluster-kafka-version
  --cluster-arn arn:aws:kafka:us-east-1:123456789012:cluster/cc-kafka-cluster/abcd1234-abcd-1234-abcd-1234abcd1234-ab
  --current-version 2.3.1
  --target-kafka-version 2.8.1

04 The output should return the update-security command request metadata:

{
	"ClusterArn": "arn:aws:kafka:us-east-1:123456789012:cluster/cc-kafka-cluster/abcd1234-abcd-1234-abcd-1234abcd1234-ab",
	"ClusterOperationArn": "arn:aws:kafka:us-east-1:123456789012:cluster/cc-kafka-cluster/abcd1234-abcd-1234-abcd-1234abcd1234-ab/123456789012"
}

05 Repeat steps no. 1 – 4 for each Amazon MSK cluster that you want to update, available in the selected AWS region.

06 Change the AWS cloud region by updating the --region command parameter value and repeat steps no. 1 – 6 to perform the Remediation process for other regions.