Enable Enhanced Monitoring for Apache Kafka Brokers

Risk Level: Medium (should be achieved)

Ensure that enhanced monitoring of Apache Kafka brokers using Amazon CloudWatch is enabled at the Amazon MSK cluster level in order to gain additional insight into the operations of your Apache Kafka clusters using an expanded set of metrics.

Reliability

Sustainability

Performance
efficiency

Cost
optimisation

Amazon Managed Streaming for Kafka (MSK) integrates with Amazon CloudWatch to collect, view, and analyze additional CloudWatch metrics for your MSK clusters. These additional metrics are automatically collected and pushed to Amazon CloudWatch in order to help you to diagnose potential issues that could limit the reliability, performance efficiency, cost optimization, and sustainability of your Amazon MSK clusters.

Audit

To determine if enhanced monitoring with Amazon CloudWatch is enabled for your MSK clusters, perform the following operations:

Using AWS Console

01 Sign in to the AWS Management Console.

02 Navigate to Amazon MSK console at https://console.aws.amazon.com/msk.

03 In the main navigation panel, underMSK Clusters, choose Clusters.

04 Click on the name (link) of the cluster that you want to examine, available in the Cluster namecolumn.

05 Select the Properties tab to access the cluster properties and attributes.

06 In the Monitoring metrics section, under Amazon CloudWatch metrics for this cluster, check the Monitoring level attribute value. If the Monitoring level value is set to Basic, enhanced monitoring of Apache Kafka brokers using Amazon CloudWatch is disabled for the selected Amazon MSK cluster.

07 Repeat steps no. 4 – 6 for each Amazon Managed Streaming for Kafka (MSK) cluster available within the current AWS region.

08 Change the AWS cloud region from the navigation bar and repeat the Audit process for the other regions.

Using AWS CLI

01 Run list-clusters command (OSX/Linux/UNIX) using custom query filters to list the Amazon Resource Names (ARNs) of the Amazon MSK clusters available in the selected AWS region:

aws kafka list-clusters
  --region us-east-1
  --query 'ClusterInfoList[*].ClusterArn'

02 The command output should return an array with the requested cluster ARNs:

[
	"arn:aws:kafka:us-east-1:123456789012:cluster/cc-kafka-cluster/abcd1234-abcd-1234-abcd-1234abcd1234-ab",
	"arn:aws:kafka:us-east-1:123456789012:cluster/cc-msk-app-cluster/aabbccdd-1234-aabb-1234-aabbccddaabb-cd"
]

03 Run describe-cluster command (OSX/Linux/UNIX) using the ARN of the Amazon MSK cluster that you want to examine as the identifier parameter and custom query filters to determine if enhanced monitoring is enabled for the selected cluster:

aws kafka describe-cluster
  --region us-east-1
  --cluster-arn arn:aws:kafka:us-east-1:123456789012:cluster/cc-kafka-cluster/abcd1234-abcd-1234-abcd-1234abcd1234-ab
  --query 'ClusterInfo.EnhancedMonitoring'

04 The command output should return the enhanced monitoring level configured for the specified cluster:

"DEFAULT"

If the describe-cluster command output returns "DEFAULT", as shown in the output example above, enhanced monitoring of Apache Kafka brokers using Amazon CloudWatch is disabled for the selected Amazon MSK cluster.

05 Repeat steps no. 3 and 4 for each Amazon Managed Streaming for Kafka (MSK) cluster provisioned in the selected AWS region.

06 Change the AWS cloud region by updating the --region command parameter value and repeat steps no. 1 – 5 to perform the Audit process for other regions.

Remediation / Resolution

To enable enhanced monitoring of Apache Kafka brokers using Amazon CloudWatch, perform the following operations:

Using AWS CloudFormation

01 CloudFormation template (JSON):

{
	"AWSTemplateFormatVersion": "2010-09-09",
	"Description": "Enable Enhanced Monitoring for Brokers",
	"Resources": {
		"MSKCluster": {
			"Type": "AWS::MSK::Cluster",
			"Properties": {
				"ClusterName": "cc-production-msk-cluster",
				"KafkaVersion": "3.4.0",
				"NumberOfBrokerNodes": 2,
				"BrokerNodeGroupInfo": {
					"BrokerAZDistribution": "DEFAULT",
					"ClientSubnets": [
						"subnet-0abcd1234abcd1234",
						"subnet-01234abcd1234abcd"
					],
					"InstanceType": "kafka.m5.large",
					"SecurityGroups": [
						"sg-0abcd1234abcd1234"
					],
					"StorageInfo": {
						"EbsStorageInfo": {
							"VolumeSize": 500
						}
					}
				},
				"EncryptionInfo": {
					"EncryptionInTransit": {
						"ClientBroker": "TLS",
						"InCluster": true
					},
					"EncryptionAtRest": {
						"DataVolumeKMSKeyId": "arn:aws:kms:us-east-1:123456789012:key/1234abcd-1234-abcd-1234-abcd1234abcd"
					}
				},
				"EnhancedMonitoring": "PER_BROKER"
			}
		}
	}
}

02 CloudFormation template (YAML):

AWSTemplateFormatVersion: '2010-09-09'
	Description: Enable Enhanced Monitoring for Brokers
	Resources:
	MSKCluster:
		Type: AWS::MSK::Cluster
		Properties:
		ClusterName: cc-production-msk-cluster
		KafkaVersion: 3.4.0
		NumberOfBrokerNodes: 2
		BrokerNodeGroupInfo:
			BrokerAZDistribution: DEFAULT
			ClientSubnets:
			- subnet-0abcd1234abcd1234
			- subnet-01234abcd1234abcd
			InstanceType: kafka.m5.large
			SecurityGroups:
			- sg-0abcd1234abcd1234
			StorageInfo:
			EbsStorageInfo:
				VolumeSize: 500
		EncryptionInfo:
			EncryptionInTransit:
			ClientBroker: TLS
			InCluster: true
			EncryptionAtRest:
			DataVolumeKMSKeyId: arn:aws:kms:us-east-1:123456789012:key/1234abcd-1234-abcd-1234-abcd1234abcd
		EnhancedMonitoring: PER_BROKER

Using Terraform (AWS Provider)

01 Terraform configuration file (.tf):

terraform {
	required_providers {
		aws = {
			source  = "hashicorp/aws"
			version = "~> 4.0"
		}
	}

	required_version = ">= 0.14.9"
}

provider "aws" {
	profile = "default"
	region  = "us-east-1"
}

resource "aws_msk_cluster" "msk-cluster" {
	cluster_name           = "cc-production-msk-cluster"
	kafka_version          = "3.4.0"
	number_of_broker_nodes = 2

	broker_node_group_info {
		instance_type  = "kafka.m5.large"
		client_subnets = ["subnet-0abcd1234abcd1234","subnet-01234abcd1234abcd"]
		storage_info {
			ebs_storage_info {
			volume_size = 500
			}
		}
		security_groups = ["sg-0abcd1234abcd1234"]
	}

	encryption_info {
		encryption_in_transit {
			client_broker = "TLS"
			in_cluster    = true
		}
		encryption_at_rest_kms_key_arn = "arn:aws:kms:us-east-1:123456789012:key/1234abcd-1234-abcd-1234-abcd1234abcd"
	}

	# Enable Enhanced Monitoring for Brokers
	enhanced_monitoring = "PER_BROKER"
}

Using AWS Console

01 Sign in to the AWS Management Console.

02 Navigate to Amazon MSK console at https://console.aws.amazon.com/msk.

03 In the main navigation panel, underMSK Clusters, choose Clusters.

04 Click on the name (link) of the cluster that you want to reconfigure.

05 Select the Properties tab and choose Edit from the Monitoring metrics section to modify the monitoring configuration available for the selected cluster.

06 On the Edit monitoring for <cluster-name> page, in the Monitoring section, select Enhanced broker-level monitoring to enable enhanced monitoring of Apache Kafka brokers using Amazon CloudWatch. Choose Save changes to apply the configuration changes.

07 Repeat steps no. 4 – 6 to enable enhanced monitoring of Apache Kafka brokers for each Amazon MSK cluster available within the current AWS region.

08 Change the AWS cloud region from the navigation bar and repeat the Remediation process for the other regions.

Using AWS CLI

01 Run update-monitoring command (OSX/Linux/UNIX) using the ARN of the Amazon MSK cluster that you want to reconfigure as the identifier parameter, to enable enhanced monitoring of Apache Kafka brokers using Amazon CloudWatch, for the selected cluster:

aws kafka update-monitoring
  --region us-east-1
  --cluster-arn arn:aws:kafka:us-east-1:123456789012:cluster/cc-kafka-cluster/abcd1234-abcd-1234-abcd-1234abcd1234-ab
  --current-version ABCDABCDABCDA
  --enhanced-monitoring PER_BROKER

02 The output should return the update-monitoring command request metadata:

{
	"ClusterArn": "arn:aws:kafka:us-east-1:123456789012:cluster/cc-kafka-cluster/abcd1234-abcd-1234-abcd-1234abcd1234-ab",
	"ClusterOperationArn": "arn:aws:kafka:us-east-1:123456789012:cluster-operation/cc-kafka-cluster/abcd1234-abcd-1234-abcd-1234abcd1234-ab/abcd1234-abcd-1234-abcd-1234abcd1234"
}

03 Repeat steps no. 1 and 2 to enable enhanced monitoring of Apache Kafka brokers for each Amazon MSK cluster provisioned in the selected AWS region.

04 Change the AWS cloud region by updating the --region command parameter value and repeat steps no. 1 – 3 to perform the Remediation process for other regions.

References

AWS Documentation
Amazon MSK FAQs
Monitoring an Amazon MSK cluster
Amazon MSK metrics for monitoring with CloudWatch
Viewing Amazon MSK metrics using CloudWatch

AWS Command Line Interface (CLI) Documentation
kafka
list-clusters
describe-cluster
update-monitoring

Publication date Jan 18, 2024

Audit

Using AWS Console

Using AWS CLI

Remediation / Resolution

Using AWS CloudFormation

Using Terraform (AWS Provider)

Using AWS Console

Using AWS CLI

References

Related MSK rules