EMR In-Transit and At-Rest Encryption

01 Sign in to the AWS Management Console.

02 Navigate to EMR dashboard at https://console.aws.amazon.com/elasticmapreduce/.

03 In the left navigation panel, under Amazon EMR, click Clusters to access your AWS EMR clusters.

04 Select the EMR cluster that you want to examine, then click on the View details button from the dashboard top menu.

05 On the selected cluster configuration details page, click on the Summary tab to access the EMR cluster configuration details.

06 In the Security and access section, search for Security configuration attribute. The Security configuration attribute value references the name of the security configuration that defines data encryption and authentication settings for your Amazon EMR cluster. If there is no Security configuration attribute listed within Security and access section, the selected Amazon Elastic MapReduce (EMR) cluster is not associated with an EMR security configuration, therefore the selected cluster does not have in-transit and at-rest encryption enabled.

07 Repeat steps no. 4 – 6 to verify in-transit and at-rest encryption status for other Amazon EMR clusters provisioned in the current region.

08 Change the AWS region from the navigation bar and repeat the audit process for other regions.

01 Run list-clusters command (OSX/Linux/UNIX) using custom query filters to list the identifiers (IDs) of all the active Amazon EMR clusters available in the selected region:

aws emr list-clusters
    --region us-east-1
    --active
    --output table
    --query 'Clusters[*].Id'

02 The command output should return a table with the requested cluster IDs:

---------------------
|   ListClusters    |
+-------------------+
|  j-AAABBBCCCDDDE  |
|  j-BBBCCCDDDEEEF  |
|  j-CCCDDDEEEFFFG  |
+-------------------+ 

03 Run describe-cluster command (OSX/Linux/UNIX) using the ID of the cluster that you want to examine as identifier and custom query filters to return the name of the EMR security configuration that defines data encryption and authentication settings for the selected AWS EMR cluster:

aws emr describe-cluster
    --region us-east-1
    --cluster-id j-AAABBBCCCDDDE
    --query 'Cluster.SecurityConfiguration'

04 The command output should return the name of the security configuration associated with the selected EMR cluster, otherwise, the command returns null for the request. If the describe-cluster command output returns null, the selected Amazon Elastic MapReduce (EMR) cluster is not associated with an EMR security configuration, therefore the selected cluster does not have in-transit and at-rest encryption enabled.

05 Repeat step no. 3 and 4 to verify in-transit and at-rest encryption status for other Amazon EMR clusters launched in the current region.

06 Change the AWS region by updating the --region command parameter value and repeat steps no. 1 - 5 to perform the audit process for other regions.

01 Login to the AWS Management Console.

02 Navigate to EMR dashboard at https://console.aws.amazon.com/elasticmapreduce/.

03 In the navigation panel, click Security configurations to access the EMR security configurations page, then click Create to start the setup process.

04 On the Create security configuration page, perform the following actions:

1. Within Name box, provide a unique name for the new EMR security configuration.
2. Select At-rest encryption checkbox to enable at-rest encryption for data stored within the cluster file system. To configure data-at-rest encryption, perform the following:
   • Under S3 encryption, choose a value from Encryption mode dropdown list to determine how AWS EMR encrypts Amazon S3 data with EMRFS. To select the right encryption mode to encrypt your EMR data, see the official AWS documentation page.
   • Under Local disk encryption, from Key provider type dropdown list, choose the default AWS KMS key or custom key provider to use for encrypting the volumes attached to EMR cluster instances.
3. Select In-transit encryption checkbox to enable the open-source TLS encryption features for EMR in-transit data. To configure in-transit encryption, perform the following:
   • Under TLS certificate provider, from Certificate provider type dropdown list, choose PEM to use PEM files that you provide in a zip file. Two artifacts should be available in your zip file: a PrivateKey.pem file and a CertificateChain.pem file (see Providing Certificates for In-Transit Data Encryption with Amazon EMR Encryption page for more details). Within S3 object box, enter the location of the zip file that contains your certificate PEM files. If you choose Custom from the Certificate provider type dropdown list, you need to specify a custom certificate provider and specify the AWS S3 location of the custom certificate-provider file. Within Certificate provider class box, type the full name of a class declared in your EMR application that implements the TLSArtifactsProvider interface.
4. Click Create to create your new AWS EMR security configuration.

05 In the navigation panel, under Amazon EMR, click Clusters to access your AWS EMR clusters page.

06 Select the EMR cluster that you want to relaunch (see Audit section part I to identify the right resource) then click on the Clone button from the dashboard top menu.

07 Inside the Cloning <your-cluster-ID> dialog box, choose Yes to include the steps from the source cluster into the cloned (destination) cluster. Click Clone to start the cloning process.

08 On the Create Cluster page, in the Security Options section, click Authentication and encryption to expand the setting panel and access the security configuration.

09 Select the name of the security configuration created at step no. 4 from the Security configuration dropdown list, then click Create Cluster to provision your new and encrypted Amazon EMR cluster.

10 Once you have moved the existing data and verified that your new EMR cluster is working 100% with the new security configuration, terminate the source cluster in order to stop incurring charges for it. To terminate the unencrypted AWS EMR cluster, perform the following:

1. Go back to the navigation panel and under Amazon EMR choose Cluster list.
2. Select the AWS EMR cluster that you want to shut down.
3. Click on the Terminate button from the dashboard top menu.
4. In the Terminate clusters confirmation box, review the original cluster details then click Terminate.

11 Repeat steps no. 4 - 10 to enable in-transit and at-rest encryption for other Amazon EMR clusters provisioned in the current region.

12 Change the AWS region from the navigation bar and repeat the entire process for other regions.

01 Run create-security-configuration command (OSX/Linux/UNIX) to create the required AWS EMR security configuration. The following command example creates an EMR security configuration named "cc-emr-security-config" with in-transit encryption enabled with PEM for certificate provider (with PrivateKey.pem and CertificateChain.pem files available at s3://cc-config-store/artifacts/cc-certificates.zip), and at-rest encryption enabled with SSE-S3 for S3 encryption and AWS-KMS (identified by the ARN arn:aws:kms:us-east-1:123456789012:key/aaaabbbb-cccc-dddd-eeee-aaaabbbbcccc) for local disk key provider:

aws emr create-security-configuration
    --region us-east-1
    --name "cc-emr-security-config"
    --security-configuration '{
    "EncryptionConfiguration": {
        "EnableInTransitEncryption" : true,
        "InTransitEncryptionConfiguration" : {
            "TLSCertificateConfiguration" : {
                "CertificateProviderType" : "PEM",
                "S3Object" : "s3://cc-config-store/artifacts/cc-certificates.zip"
            }
        },
        "EnableAtRestEncryption" : true,
        "AtRestEncryptionConfiguration" : {
            "S3EncryptionConfiguration" : {
                "EncryptionMode" : "SSE-S3"
            },
            "LocalDiskEncryptionConfiguration" : {
                "EncryptionKeyProviderType" : "AwsKms",
                "AwsKmsKey" : "arn:aws:kms:us-east-1:123456789012:key/aaaabbbb-cccc-dddd-eeee-aaaabbbbcccc"
            }
        }
    }
}'

02 The command output should return the metadata for the new EMR security configuration:

{
    "CreationDateTime": 1512587891.433,
    "Name": "cc-emr-security-config"
} 

03 Get the configuration details from the source EMR cluster, required for the next step. Run describe-cluster command (OSX/Linux/UNIX) using the ID of the cluster that you want to re-create (see Audit section part II to identify the right resource), to describe its configuration details:

aws emr describe-cluster
    --region us-east-1
    --cluster-id j-AAABBBCCCDDDE

04 The command output should return the source EMR cluster configuration information:

{
    "Cluster": {
        "Name": "cc-emr-cluster",
        "ServiceRole": "EMR_DefaultRole",
        "TerminationProtected": false,
        "ReleaseLabel": "emr-5.0.0",
        "NormalizedInstanceHours": 6,
        "InstanceCollectionType": "INSTANCE_GROUP",
 
        ...
 
        "ScaleDownBehavior": "TERMINATE_AT_TASK_COMPLETION",
        "VisibleToAllUsers": true,
        "BootstrapActions": [],
        "AutoTerminate": false,
        "Id": "j-AAABBBCCCDDDE"
    }
}

05 Run create-cluster command (OSX/Linux/UNIX) using the configuration details returned at the previous step as values for the required parameters to re-create the existing EMR cluster and enable in-transit and at-rest encryption with the EMR security configuration created at step no. 1:

aws emr create-cluster
    --region us-east-1
    --name cc-emr-encrypted-cluster
    --release-label emr-5.0.0
    --instance-groups InstanceGroupType=MASTER,InstanceCount=1,InstanceType=c4.xlarge InstanceGroupType=CORE,InstanceCount=2,InstanceType=c4.xlarge
    --service-role EMR_DefaultRole
    --ec2-attributes KeyName=SSHAccessKey,InstanceProfile=EMR_EC2_DefaultRole,EmrManagedMasterSecurityGroup=sg-aaaabbbb,EmrManagedSlaveSecurityGroup=sg-ddddeeee,AvailabilityZone=us-east-1a,SubnetId=subnet-aaaabbbb
    --visible-to-all-users
    --no-auto-terminate
    --no-termination-protected
    --security-configuration cc-emr-security-config

06 The command output should return the ID of the newly created EMR cluster:

{
    "ClusterId": "j-DDDEEEFFFGGGH"
}

07 Once the source cluster data is migrated and you have verified that your new EMR cluster is working 100% with the new security configuration, terminate the source cluster to stop incurring charges for it. To shut down the source EMR cluster, run terminate-clusters command (OSX/Linux/UNIX) using its ID as identifier (the command does not produce an output):

aws emr terminate-clusters
    --region us-east-1
    --cluster-ids j-AAABBBCCCDDDE

08 Repeat steps no. 1 – 7 to enable in-transit and at-rest encryption for other Amazon EMR clusters launched in the current region.

09 Change the AWS region by updating the --region command parameter value and repeat the process for other regions.