- Knowledge Base
- Amazon Web Services
- AWS Macie v2
- Amazon Macie Sensitive Data Repository
Ensure that the Amazon Macie service is configured to store data discovery results in an Amazon S3 bucket to discover, classify, and protect sensitive data within AWS cloud. Amazon Macie will scan your S3 buckets to identify sensitive information, store the results to a repository bucket, and bring this data to your attention in order to analyze access patterns and user behavior, and prevent data leakage.
This rule resolution is part of the Conformity solution.
To access the sensitive data discovery results generated by Amazon Macie and enable long-term storage and retention of these results, an Amazon S3 bucket must be configured as a data repository within 30 days of enabling Amazon Macie. To follow AWS security best practices, the Amazon S3 bucket configured as a data repository must be encrypted using an AWS KMS key. Once the repository bucket is configured, Macie writes your sensitive data discovery results to JSON Lines (.jsonl) files, which it adds to the S3 bucket as GNU Zip (.gz) files. The configured S3 bucket can then serve as a definitive, long-term repository for all of your Macie data discovery results.
Audit
To determine if the Amazon Macie service has been configured to store data discovery results in an S3 bucket (also known as Macie data repository), perform the following operations:
Using AWS Console
01 Sign in to the AWS Management Console.
02 Navigate to Amazon Macie console at https://console.aws.amazon.com/macie/.
03 In the left navigation panel, under Settings, choose Discovery results.
04 In the Repository for sensitive data discovery results section, check the S3 bucket attribute value to determine the name of the Amazon S3 bucket configured as the repository for Macie data discovery results. If the S3 bucket attribute is not listed in this section and the following message is displayed: "Configure an S3 bucket for long-term retention of your sensitive data discovery results.", Amazon Macie is not configured to save sensitive data findings to an S3 bucket within the current AWS region.
05 Change the AWS cloud region from the console navigation bar and repeat the Audit process for other regions.
Using AWS CLI
01 Run get-classification-export-configuration command (OSX/Linux/UNIX) with custom query filters to describe the name of the Amazon S3 bucket configured to store Macie data classification (discovery) results in the selected AWS cloud region:
aws macie2 get-classification-export-configuration --region us-east-1 --query 'configuration.s3Destination.bucketName'
02 The command output should return the name of the S3 bucket configured as Macie data repository:
null
If the get-classification-export-configuration command output returns null, as shown in the example above, Amazon Macie is not configured to save sensitive data findings to an S3 bucket within the selected AWS region.
03 Change the AWS cloud region by updating the --region command parameter value and repeat the Audit process for other regions.
Remediation / Resolution
To ensure long-term retention of sensitive data discovery results, an Amazon S3 bucket must be configured as a data repository within Amazon Macie. To create and configure the data repository bucket, perform the following operations:
Using AWS Console
01 Sign in to the AWS Management Console.
02 To create the KMS key required to encrypt the repository data, navigate to Amazon KMS console at https://console.aws.amazon.com/kms/.
03 In the navigation panel, under Key Management Service (KMS), select Customer managed keys.
04 Choose the Create Key button from the console top menu to initiate the KMS key setup process.
05 For Step 1 Configure key, perform the following actions:
- Choose Symmetric from the Key type section. A symmetric key is a single encryption key that can be used for both encrypt and decrypt operations.
- Under Advanced options, for Key material origin, select KMS as the source of the key material within the key.
- Under Advanced options, for Regionality, select whether to allow the new key to be replicated into other AWS regions.
- Choose Next to continue.
06 For Step 2 Add labels, type a unique name (alias) for your new master key in the Alias box and provide a short description for the key in Description – optional box. (Optional) Use the Add tag button to create tags in order to categorize and identify your KMS key. Choose Next to continue the setup process.
07 For Step 3 Define key administrative permissions, choose which IAM users and/or roles can administer your new key from the Key administrators section. You may need to add additional permissions for the users or roles to administer the key from the AWS console. For Key deletion, select Allow key administrators to delete this key. Choose Next to continue.
08 For Step 4 Define key usage permissions, within This account section, select which IAM users and/or roles can use the new Customer Master Key (CMK) for cryptographic operations. (Optional) In the Other AWS accounts section, choose Add another AWS account and enter an external AWS account ID in order to specify the external AWS account that can use the new key to encrypt and decrypt your Macie data discovery results. The owners of the external AWS accounts must also provide access to this key by creating appropriate policies for their IAM users. Choose Next to continue.
09 For Step 5 Review, review the policy available in the Key policy section, then choose Finish to create your new Customer Master Key (CMK). Once the key is successfully created, the Amazon KMS console will display the following confirmation message: "Success. Your customer master key was created with alias key-alias
and key ID key-id
".
10 Click on the name (alias) of the newly created KMS key and choose the Key policy tab.
11 In the Key policy section, choose Edit, add the following statement to the existing policy, then choose Save changes to apply the changes:
{ "Sid": "Allow Macie to use the key", "Effect": "Allow", "Principal": { "Service": "macie.amazonaws.com" }, "Action": [ "kms:GenerateDataKey", "kms:Encrypt" ], "Resource": "*" }
12 Navigate to Amazon Macie console at https://console.aws.amazon.com/macie/.
13 In the left navigation panel, under Settings, choose Discovery results.
14 To set up a repository for sensitive data discovery results, under Settings, choose Discovery results.
15 Select Configure now to configure an Amazon S3 bucket for long-term retention of your data discovery results.
16 In the Repository for sensitive data discovery results section, choose Configure now to initiate the setup process.
17 On the Repository for sensitive data discovery results setup page, perform the following actions:
- Choose Create bucket to create a new Amazon S3 bucket.
- Provide a unique name for the new S3 bucket in the Create a bucket box.
- Choose Advanced and set Block all public access to Yes to block anonymous access to the bucket. (Optional) You can also provide a location prefix in the Data discovery result prefix box.
- For KMS encryption, choose Select a key from your account, and select the name (alias) of the KMS key that you want to use for data encryption, from the KMS key alias dropdown list.
- Choose Save to apply the configuration changes.
18 Repeat steps no. 2 – 17 to perform the Remediation process for each supported AWS region.
Using AWS CLI
01 Define the policy that enables the selected IAM users and/or roles to manage the new KMS key, and to encrypt/decrypt your Amazon Macie repository data using the KMS API. Create a new policy document (JSON format), name the file macie-repository-key-policy.json, and paste the following content (replace the highlighted details, i.e. the ARNs for the IAM users and/or roles, with your own details):
{ "Id": "macie-data-repo-key-policy", "Version": "2012-10-17", "Statement": [ { "Sid": "Enable IAM User Permissions", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::aws-account-id
:root" }, "Action": "kms:*", "Resource": "*" }, { "Sid": "Allow Amazon Macie to use the key", "Effect": "Allow", "Principal": { "Service": "macie.amazonaws.com" }, "Action": [ "kms:GenerateDataKey", "kms:Encrypt" ], "Resource": "*" }, { "Sid": "Allow access for Key Administrators", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::aws-account-id
:role/role-name
" }, "Action": [ "kms:Create*", "kms:Describe*", "kms:Enable*", "kms:List*", "kms:Put*", "kms:Update*", "kms:Revoke*", "kms:Disable*", "kms:Get*", "kms:Delete*", "kms:TagResource", "kms:UntagResource", "kms:ScheduleKeyDeletion", "kms:CancelKeyDeletion" ], "Resource": "*" }, { "Sid": "Allow use of the key", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::aws-account-id
:role/role-name
" }, "Action": [ "kms:Encrypt", "kms:Decrypt", "kms:ReEncrypt*", "kms:GenerateDataKey*", "kms:DescribeKey" ], "Resource": "*" }, { "Sid": "Allow attachment of persistent resources", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::aws-account-id
:role/role-name
" }, "Action": [ "kms:CreateGrant", "kms:ListGrants", "kms:RevokeGrant" ], "Resource": "*", "Condition": { "Bool": { "kms:GrantIsForAWSResource": "true" } } } ] }
02 Run create-key command (OSX/Linux/UNIX) using the policy document created at the previous step (i.e. macie-repository-key-policy.json) as value for the --policy parameter, to create your new KMS key:
aws kms create-key --region us-east-1 --description 'KMS key for encrypting Amazon Macie repository data' --policy file://macie-repository-key-policy.json --query 'KeyMetadata.Arn'
03 The command output should return the ARN of the new KMS key:
"arn:aws:kms:us-east-1:123456789012:key/abcdabcd-1234-abcd-1234-abcd1234abcd"
04 Run create-alias command (OSX/Linux/UNIX) using the key ARN returned at the previous step to attach an alias to the new key. The alias must start with the prefix "alias/" (the command should not produce an output):
aws kms create-alias --region us-east-1 --alias-name alias/MacieDataEncryptionKey --target-key-id arn:aws:kms:us-east-1:123456789012:key/abcdabcd-1234-abcd-1234-abcd1234abcd
05 Run create-bucket command (OSX/Linux/UNIX) to create the Amazon S3 bucket that will serve as the repository for your Macie data discovery results within the selected AWS region:
aws s3api create-bucket --region us-east-1 --bucket cc-discovery-results-repository --acl private
Note: for creating S3 buckets outside the us-east-1 region, you will need to define the region as us-east-1 and the LocationConstraint as your chosen region:
aws s3api create-bucket --region us-east-1 --bucket cc-discovery-results-repository --acl private --create-bucket-configuration LocationConstraint=eu-west-1
06 The command output should return the name of the newly created S3 bucket:
{ "Location": "/cc-discovery-results-repository" }
07 Run put-public-access-block command (OSX/Linux/UNIX) to enable the S3 Public Access Block feature for the new S3 bucket (the command should not produce an output):
aws s3api put-public-access-block --region us-east-1 --bucket cc-discovery-results-repository --public-access-block-configuration BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true
08 Define the access policy for the newly created Amazon S3 bucket. Save the following bucket policy to a JSON file named repository-bucket-policy.json (replace the highlighted details, i.e. the ARNs for the bucket and the KMS key, with your own details):
{ "Version": "2012-10-17", "Statement": [ { "Sid": "Deny non-HTTPS access", "Effect": "Deny", "Principal": "*", "Action": "s3:*", "Resource": "arn:aws:s3:::cc-discovery-results-repository
/*", "Condition": { "Bool": { "aws:SecureTransport": "false" } } }, { "Sid": "Deny incorrect encryption header. This is optional", "Effect": "Deny", "Principal": { "Service": "macie.amazonaws.com" }, "Action": "s3:PutObject", "Resource": "arn:aws:s3:::cc-discovery-results-repository
/*", "Condition": { "StringNotEquals": { "s3:x-amz-server-side-encryption-aws-kms-key-id": "arn:aws:kms:us-east-1:123456789012:key/abcdabcd-1234-abcd-1234-abcd1234abcd
" } } }, { "Sid": "Deny unencrypted object uploads. This is optional", "Effect": "Deny", "Principal": { "Service": "macie.amazonaws.com" }, "Action": "s3:PutObject", "Resource": "arn:aws:s3:::cc-discovery-results-repository
/*", "Condition": { "StringNotEquals": { "s3:x-amz-server-side-encryption": "aws:kms" } } }, { "Sid": "Allow Macie to upload objects to the bucket", "Effect": "Allow", "Principal": { "Service": "macie.amazonaws.com" }, "Action": "s3:PutObject", "Resource": "arn:aws:s3:::cc-discovery-results-repository
/*" }, { "Sid": "Allow Macie to use the getBucketLocation operation", "Effect": "Allow", "Principal": { "Service": "macie.amazonaws.com" }, "Action": "s3:GetBucketLocation", "Resource": "arn:aws:s3:::cc-discovery-results-repository
" } ] }
09 Run put-bucket-policy command (OSX/Linux/UNIX) to attach the bucket policy defined at the previous step to your new Amazon S3 bucket (the command does not produce an output):
aws s3api put-bucket-policy --region us-east-1 --bucket cc-discovery-results-repository --policy file://repository-bucket-policy.json
10 Run put-classification-export-configuration command (OSX/Linux/UNIX) to update the configuration settings for storing Macie data discovery results and set up the Amazon S3 bucket created at the previous steps as data repository in the selected AWS region:
aws macie2 put-classification-export-configuration --region us-east-1 --configuration "s3Destination={bucketName=cc-discovery-results-repository,kmsKeyArn=arn:aws:kms:us-east-1:123456789012:key/abcdabcd-1234-abcd-1234-abcd1234abcd}"
11 The command output should return the updated configuration settings:
{ "configuration": { "s3Destination": { "bucketName": "cc-discovery-results-repository", "kmsKeyArn": "arn:aws:kms:us-east-1:123456789012:key/abcdabcd-1234-abcd-1234-abcd1234abcd" } } }
12 Change the AWS cloud region by updating the --region command parameter value and repeat the Remediation process for each supported AWS region.
References
- AWS Documentation
- What Is Amazon Macie Classic?
- Concepts and Terminology
- Getting started with Amazon Macie
- Storing and retaining sensitive data discovery results with Amazon Macie
- AWS Command Line Interface (CLI) Documentation
- macie2
- get-classification-export-configuration
- put-classification-export-configuration
- kms
- create-key
- create-alias
- s3api
- create-bucket
- put-public-access-block
- put-bucket-policy