01 Before you can set up and manage your Customer-Managed Keys (CMKs), you must create a key ring to store the CMKs. Run kms keyrings create command (Windows/macOS/Linux) to create a new Cloud KMS key ring in the specified location. If the CMKs created later within this key ring will be used to encrypt/decrypt resources in a given region, select that region as the key ring location:
gcloud kms keyrings create cc-cloud-sql-key-ring
--location=us
--project=cc-bigdata-project-123123
--format="table(name)"
02 The command output should return the identifier (name) of the newly created key ring:
NAME
projects/cc-bigdata-project-123123/locations/us/keyRings/cc-dataproc-key-ring
03 Run kms keys create command (Windows/macOS/Linux) to create a new Cloud KMS Customer-Managed Key (CMK) within the KMS key ring created at the previous steps:
gcloud kms keys create cc-dataproc-cluster-cmk
--location=us-central1
--keyring=cc-dataproc-key-ring
--purpose=encryption
--protection-level=software
--rotation-period=90d
--next-rotation-time=2020-9-15T10:00:00.0000Z
--format="table(name)"
04 The command output should return the name of the new Customer-Managed Key (CMK):
NAME
projects/cc-bigdata-project-123123/locations/us-central1/keyRings/cc-dataproc-key-ring/cryptoKeys/cc-dataproc-cluster-cmk
05 Run projects add-iam-policy-binding command (Windows/macOS/Linux) to assign the Cloud KMS "CryptoKey Encrypter/Decrypter" role to the appropriate service account. Replace <kms-project-id> with the ID of the Google Cloud project where the Customer-Managed Keys are provisioned, and replace <dataproc-project-number> with the project number of the Google Cloud project that is running your Dataproc clusters:
gcloud projects add-iam-policy-binding <kms-project-id>
--member serviceAccount:service-<dataproc-project-number>@compute-system.iam.gserviceaccount.com
--role roles/cloudkms.cryptoKeyEncrypterDecrypter
06 The command output should return the updated IAM policy (YAML format):
Updated IAM policy for project <kms-project-id>.
bindings:
- members:
- serviceAccount:service-<project-number>@compute-system.iam.gserviceaccount.com
role: roles/cloudkms.cryptoKeyEncrypterDecrypter
- members:
- user:admin@cloudconformity.com
role: roles/owner
etag: abcdabcdabcd
version: 1
07 Run dataproc clusters describe command (Windows/macOS/Linux) using the name of the Google Cloud Dataproc cluster that you want to examine as identifier parameter and custom query filters to describe the configuration metadata available for the selected cluster:
gcloud dataproc clusters describe cc-dataproc-prod-cluster
--region=us-central1
--format=json
08 The command output should return the requested configuration metadata:
{
"clusterName": "cc-dataproc-prod-cluster",
"config": {
"configBucket": "dataproc-staging-us-central1-123456789012-abcdabcd",
"masterConfig": {
"diskConfig": {
"bootDiskSizeGb": 500,
"bootDiskType": "pd-standard"
},
"machineTypeUri": "https://www.googleapis.com/compute/v1/projects/cc-bigdata-project-123123/zones/us-central1-a/machineTypes/n1-standard-1",
"minCpuPlatform": "AUTOMATIC",
},
...
"tempBucket": "dataproc-temp-us-central1-6123456789012-abcdabcd"
},
"projectId": "cc-bigdata-project-123123",
"status": {
"state": "RUNNING",
"stateStartTime": "2020-07-19T08:20:00.000Z"
},
"statusHistory": [
{
"state": "CREATING",
"stateStartTime": "2020-07-19T08:20:00.000Z"
}
]
}
09 Run dataproc clusters create command (Windows/macOS/Linux) using the information returned at the previous step as configuration data for the command parameters, to create a new Google Cloud Dataproc cluster, encrypted with the Customer-Managed Key (CMK) created at step no. 3:
gcloud dataproc clusters create cc-encrypted-dataproc-cluster
--region=us-central1
—-project=cc-bigdata-project-123123
--single-node
--master-machine-type=n1-standard-1
--master-boot-disk-size=500GB
--master-boot-disk-type=pd-standard
--gce-pd-kms-key=projects/cc-bigdata-project-123123/locations/us-central1/keyRings/cc-dataproc-key-ring/cryptoKeys/cc-dataproc-cluster-cmk
10 The command output should return the metadata (region and URL) for the newly created Dataproc cluster:
Waiting for cluster creation operation...done.
Created [https://dataproc.googleapis.com/v1/projects/cc-bigdata-project-123123/regions/us-central1/clusters/cc-encrypted-dataproc-cluster] Cluster placed in zone [us-central1-c].
11 If required, migrate the source cluster data to the newly created (target) cluster.
12 Update your application(s) to reference the new Google Cloud Dataproc cluster.
13 Once the new cluster is operating successfully, you can remove the source cluster in order to stop adding charges to your Google Cloud bill. Run dataproc clusters delete command (Windows/macOS/Linux) using the name of the resource that you want to remove as identifier parameter (see Audit section part II to identify the right cluster), to delete the specified Dataproc cluster:
gcloud dataproc clusters delete cc-dataproc-prod-cluster
--region=us-central1
14 Type Y to confirm the resource removal. All the cluster disks will be permanently deleted, therefore make sure that your data has been successfully exported to the new cluster before removal:
The cluster 'cc-dataproc-prod-cluster' and all attached disks will be deleted.
Do you want to continue (Y/n)? Y
15 The output should return the dataproc clusters delete command request status:
Waiting for cluster deletion operation...done.
Deleted [https://dataproc.googleapis.com/v1/projects/cc-bigdata-project-123123/regions/us-central1/clusters/cc-dataproc-prod-cluster].
16 Repeat steps no. 7 – 15 to enable encryption at rest with Customer-Managed Keys (CMKs) for other Google Cloud Dataproc clusters provisioned in the selected project
17 Repeat steps no. 1 – 16 for each GCP project deployed in your Google Cloud account.
Security 