EMR Desired Instance Type

Risk Level: Medium (should be achieved)

Rule ID: EMR-004

Determine if the AWS Elastic MapReduce (EMR) cluster instances (master and core instances) provisioned in your AWS account have the desired instance type established by your organization based on the workload deployed. Cloud Conformity provides you with the ability to define the desired instance types based on your workload requirements upon enabling this rule.

This rule can help you with the following compliance standards:

APRA
MAS

For further details on compliance standards supported by Conformity, see here.

This rule can help you work with the AWS Well-Architected Framework.

This rule resolution is part of the Conformity Security & Compliance tool for AWS.

Security

Cost
optimisation

Setting limits for the type of Amazon EMR instances provisioned in your AWS account will help you address organizational__compliance requirements and prevent unexpected charges on your monthly AWS bill.

Note 1: You can also limit your EMR__cluster instances to the desired types using AWS Organizations service by implementing your own Service Control Policy on the master account. A Service Control Policy (SCP) is a type of policy that you can use to manage your organization. SCPs enable you to restrict what resources, services and actions the users, groups, and roles in those AWS accounts can use.
Note 2: The desired Elastic MapReduce__instance type used as example within this rule is m3.xlarge. To meet your organizational requirements, you will need to configure this rule with your own desired instance types.

Audit

To determine if Amazon EMR instances launched in your AWS account have the desired type, perform the following:

Using AWS Console

01 Sign in to the AWS Management Console.

02 Navigate to EMR dashboard at https://console.aws.amazon.com/elasticmapreduce/.

03 In the left navigation panel, under Amazon EMR, click Clusters to access your AWS EMR clusters page.

04 Select Active clusters from the filter dropdown list to return only the active EMR clusters.

05 Select the EMR cluster that you want to examine then click on the View details button from the dashboard top menu.

06 On the selected cluster configuration details page, click on the Hardware tab to expand the EMR cluster hardware panel.

07 Inside the Instance Groups section, verify the value available within the Instance type column for each instance (master and core instance) provisioned in the cluster.

08 Go back to the Clusters page and repeat steps no. 5 – 7 to determine the type of the instances provisioned by other EMR clusters within the current region.

09 Change the AWS region from the navigation bar and repeat steps no. 3 – 8 for all other regions. If the value listed in the Instance type column is not the same for all active EMR resources, the AWS Elastic MapReduce instances available in the current region were not launched using the desired type, therefore you must take action and raise an AWS support case to limit EMR instance creation only to the desired type (see Remediation/Resolution section).

10 Change the AWS region from the navigation bar and repeat steps no. 4 – 9 for all other regions.

Using AWS CLI

01 Run list-clusters command (OSX/Linux/UNIX) using custom query filters to list the identifiers (IDs) of all active Amazon EMR clusters available in the selected region:

aws emr list-clusters
  --region us-east-1
  --active
  --output table
  --query 'Clusters[*].Id'

02 The command output should return a table with the requested cluster IDs:

---------------------
|   ListClusters    |
+-------------------+
|  j-1234dddeee000  |
|  j-1234bbbccc111  |
|  j-1234eeefff222  |
+-------------------+

03 Run describe-cluster command (OSX/Linux/UNIX) using the ID of the cluster that you want to examine as identifier, returned at the previous step, and custom query filters to describe the instance(s) type used by the selected Amazon EMR cluster:

aws emr describe-cluster
  --region us-east-1
  --cluster-id j-1234dddeee000
  --query 'Cluster.InstanceGroups[*].[InstanceGroupType,InstanceType]'

04 The command output should return the type used for instances within the selected AWS EMR cluster. The first array returned describes the type of core instances and second array the type of master instances:

[
	[
		"CORE",
		"m4.2xlarge"
	],
	[
		"MASTER",
		"m4.2xlarge"
	]
]

05 Repeat step no. 3 and 4 to determine the type of instances provisioned by all other AWS EMR clusters, available in the current region.

06 The describe-cluster command output should return an array with the type of EMR cluster instances (core instances and master instances), available in the selected region. If the instance(s) type returned in the command output**is not the same for all existing EMR clusters, the AWS Elastic MapReduce instances available in the current region were not created using the desired instance type, therefore you must take action and build an AWS support case to limit EMR instance creation only to the required type.

07 Repeat steps no. 1 – 6 to perform the audit process for all other AWS regions.

Remediation / Resolution

To limit the new AWS Elastic MapReduce cluster instances to the desired type, create an AWS support case where you explain why you need this type of limitation. For any existing EMR clusters launched without using the desired instance type, just clone the necessary clusters and re-create them using the desired instance type.
To create the required AWS support case, perform the following actions:

Note: Creating a support case to request the necessary resource type limitation using the AWS API via Command Line Interface (CLI) is not currently supported.

Using AWS Console

01 Sign in to the AWS Management Console.

02 Navigate to AWS Support Center dashboard at https://console.aws.amazon.com/support/.

03 On the Create Case support page, perform the following:

Under Regarding, select Account and Billing Support.
Choose Other Account Issues from the Category dropdown list.
In the Subject field, enter the request subject, e.g. "Limit the creation of AWS Elastic MapReduce cluster instances to a desired type".
In the Description textbox, enter a brief description where you explain why you need to limit the provisioning of EMR instances to a specific type so that AWS support can evaluate your case faster.
From Supported Language, choose your preferred correspondence language for the current case.
Under Contact method, select a preferred contact method that AWS support team can use to respond to your request.
Click Submit to send the limit request to Amazon Web Services. A customer support representative will contact you shortly.

References

AWS Command Line Interface (CLI) Documentation
emr
list-clusters
describe-cluster

Publication date Sep 28, 2017

Audit

Using AWS Console

Using AWS CLI

Remediation / Resolution

Using AWS Console

References

Related EMR rules