Machine Learning Workspace Encryption using Customer-Managed Keys

Risk Level: High (not acceptable risk)

Rule ID: MachineLearning-004

Ensure that your Azure Machine Learning (ML) workspaces are using Customer-Managed Keys (CMKs) instead of Microsoft-managed encryption keys (i.e. default keys used by Microsoft Azure for encryption at rest), in order to have a more granular control over your Azure Machine Learning data encryption and decryption process.

This rule resolution is part of the Conformity Security & Compliance tool for Azure.

Security

Azure Machine Learning (ML) uses various Azure data storage services and compute resources for training models and performing inferences. End-to-end projects in Azure ML integrate with services like Azure Cosmos DB and Azure Blob Storage. For Azure ML, metrics and metadata are stored in an Azure Cosmos DB instance, where all data is encrypted at rest. By default, this encryption uses Microsoft-managed keys. However, you can bring your own keys (i.e. Customer-Managed Keys) to fully control who can use the encryption keys and access the encrypted data. Using Customer-Managed Keys (CMKs) also allows you to automatically update the key version used for storage encryption whenever a new version is available in the associated Azure cloud resource.

Audit

To determine if your Azure Machine Learning workspaces are encrypted using Customer Managed Keys (CMKs), perform the following actions:

Using Azure Console

01 Sign in to the Azure Management Console.

02 Navigate to Azure Machine Learning workspaces blade at https://portal.azure.com/#browse/Microsoft.MachineLearningServices%2Fworkspaces.

03 Select the Azure subscription that you want to examine from the Subscription equals all filter box, and choose Apply.

04 Click on the name (link) of the Azure Machine Learning workspace that you want to examine.

05 In the main navigation panel, select Overview, and choose JSON View under Essentials to access the workspace configuration information available in JSON format.

06 On the Resource JSON panel, check for the "encryption.keyVaultProperties.keyIdentifier" configuration attribute value in the resource configuration object, to identify the URL of the Customer-Managed Key (CMK) used for workspace data encryption. If the "encryption.keyVaultProperties.keyIdentifier" attribute is not available, encryption at rest using Customer-Managed Keys (CMKs) is not enabled for the selected Azure Machine Learning (ML) workspace.

07 Repeat steps no. 4 – 6 for each Azure Machine Learning workspace available within the selected subscription.

08 Repeat steps no. 3 – 7 for each subscription created in your Microsoft Azure cloud account.

Using Azure CLI

01 Run ml workspace list command (Windows/macOS/Linux) with custom query filters to list the name and the associated resource group for each Azure Machine Learning workspace available in the current subscription:

az ml workspace list
  --output table
  --query '[*].{name:name, resourceGroup:resourceGroup}'

02 The command output should return the requested Azure Machine Learning workspace identifiers:

Name                           ResourceGroup
------------------------       ------------------------------
tm-ml-project5-workspace       cloud-shell-storage-westeurope
tm-ml-production-workspace     cloud-shell-storage-westeurope

03 Run ml workspace show command (Windows/macOS/Linux) with the name of the Azure Machine Learning workspace that you want to examine as the identifier parameter and custom output filters to describe the URL of the Customer-Managed Key (CMK) used for workspace data encryption:

az ml workspace show
  --name tm-ml-project5-workspace
  --resource-group cloud-shell-storage-westeurope
  --query '{"KeyURL":customer_managed_key.key_uri}'

04 The command output should return the URL of the requested encryption key:

{
	"KeyURL": null
}

If the ml workspace show command output returns null for the "KeyURL" configuration attribute, as shown in the output example above, there is no Customer-Managed Key configured for your workspace, therefore, encryption at rest using Customer-Managed Keys (CMKs) is not enabled for the selected Azure Machine Learning (ML) workspace.

05 Repeat step no. 3 and 4 for each Azure Machine Learning workspace available in the selected Azure subscription.

06 Repeat steps no. 1 – 5 for each subscription created in your Microsoft Azure cloud account.

Remediation / Resolution

To enable encryption at rest for Azure Machine Learning workspaces using Customer-Managed Keys (CMKs), you must re-create your workspaces with the necessary encryption configuration, by performing the following operations:

Using Azure Console

01 Sign in to the Azure Management Console.

02 Navigate to Key vaults blade at https://portal.azure.com/#blade/HubsExtension/BrowseResource/resourceType/Microsoft.KeyVault%2Fvaults.

03 Choose Create and perform the following actions to create the Azure key vault that will store your new Customer-Managed Key (CMK):

For Basics, choose the correct subscription and resource group, provide a unique name for the new key vault, then select the Azure cloud region where the vault will be deployed, and the appropriate pricing tier. Configure the vault retention period and enable purge protection. Choose Next to continue the vault setup process.
For Access configuration, select Vault access policy for Permission model, choose Create under Access policies, and follow the setup wizard to create the policy that allows Azure Machine Learning to create, retrieve, recover, wrap, and unwrap encryption keys from the new vault. Once the access policy is configured, choose Create to create and attach it to the key vault. (Optional) Configure Azure cloud resource access under Resource access. Choose Next to continue the setup.
For Networking, configure the network access control for the new key vault. You can connect to your new key vault either publicly, via public IP addresses or service endpoints, or privately, using a private endpoint. Choose Next to continue.
For Tags, use the Name and Value fields to create tags that will help organize the identity of the key vault. Choose Review + create to validate the key vault setup.
For Review + create, review the resource configuration details, then choose Create to create your new Azure key vault.

04 Click on the name of the newly created Microsoft Azure key vault.

05 In the resource navigation panel, under Objets, select Keys, then choose Generate/Import to create the Customer-Managed Key required for Azure Machine Learning workspace encryption.

06 On the Create a key setup page, provide a unique name for the encryption key in the Name box, set Key type to RSA, RSA key size to 2048, choose an activation and/or expiration date, set the Enabled flag to Yes, then choose Create to generate your new Customer-Managed Key (CMK).

07 Once your new Customer-Managed Key is available, navigate to Azure Machine Learning workspaces blade at https://portal.azure.com/#browse/Microsoft.MachineLearningServices%2Fworkspaces.

08 Select the Azure subscription that you want to access from the Subscription equals all filter box, and choose Apply.

09 Choose Create, select New workspace, and perform the following operations to create your new workspace:

For Basics, provide the following information:
1. For Subscription, choose your Azure subscription.
2. For Resource group, select the correct resource group.
3. Provide a unique name for the workspace in the Name box.
4. For Region, select the Azure cloud region where the workspace will be deployed.
5. For Storage account, choose the storage account that is used as the default datastore for the workspace.
6. For Key vault, select the key vault used to store secrets and other sensitive information required by the workspace.
7. For Application insights, choose whether to enable the Application Insights monitoring feature. You can create a new Azure Application Insights resource or select an existing one from your subscription.
8. For Container registry, select the container registry used to register docker images used in training and deployments.
9. Choose Next : Networking to continue the workspace setup process.
For Networking, choose the type of network isolation that you need for your workspace. Choose Next : Encryption to continue the setup.
For Encryption, perform the following actions:
1. For Encryption type, choose Customer-managed keys.
2. For Key vault, choose Click to select the key, and select the key vault and the Customer-Managed Key (CMK) created earlier in the Remediation section.
3. Choose Next : Identity to continue.
For Identity, perform the following operations:
1. Choose System assigned identity for Identity type under Managed identity.
2. Select Identity-based access for Storage account access type under Storage account access.
3. Under Data impact, check the High business impact workspace setting checkbox to enable the High Business Impact (HBI) feature.
4. Choose Next : Tags to continue the setup.
For Tags, use the Name and Value fields to create tags that will help organize the identity of the workspace. Choose Next : Review + create to validate the workspace setup.
For Review + create, review the resource configuration details, then choose Create to create your new Azure Machine Learning workspace.

10 Repeat step no. 9 for each workspace that you want to re-create, available within the selected Azure subscription.

11 Repeat steps no. 2 – 10 for each subscription available in your Microsoft Azure cloud account.

Using Azure CLI

01 Run keyvault create command (Windows/macOS/Linux) to create the Microsoft Azure key vault where the required Customer-Managed Key will be placed:

az keyvault create
  --name cc-project5-vault
  --resource-group cloud-shell-storage-westeurope
  --location westeurope
  --enabled-for-deployment true
  --enabled-for-template-deployment true
  --enable-purge-protection true
  --retention-days 30
  --enable-rbac-authorization false

02 The command output should return the object ID of the new Microsoft Azure key vault:

{
	"id": "/subscriptions/abcd1234-abcd-1234-abcd-1234abcd1234/resourceGroups/cloud-shell-storage-westeurope/providers/Microsoft.KeyVault/vaults/cc-project5-vault",
	"location": "westeurope",
	"name": "cc-project5-vault",
	"properties": {
		"accessPolicies": [
			{
				"applicationId": null,
				"objectId": "abcdabcd-abcd-abcd-abcd-abcdabcdabcd",
				"permissions": {
					"certificates": [
						"all"
					],
					"keys": [
						"all"
					],
					"secrets": [
						"all"
					],
					"storage": [
						"all"
					]
				},
				"tenantId": "abcd1234-abcd-1234-abcd-1234abcd1234"
			}
		],
		"createMode": null,
		"enablePurgeProtection": true,
		"enableRbacAuthorization": false,
		"enableSoftDelete": true,
		"enabledForDeployment": true,
		"enabledForDiskEncryption": null,
		"enabledForTemplateDeployment": true,
		"hsmPoolResourceId": null,
		"networkAcls": null,
		"privateEndpointConnections": null,
		"provisioningState": "Succeeded",
		"publicNetworkAccess": "Enabled",
		"sku": {
			"family": "A",
			"name": "standard"
		},
		"softDeleteRetentionInDays": 30,
		"tenantId": "abcd1234-abcd-1234-abcd-1234abcd1234",
		"vaultUri": "https://cc-project5-vault.vault.azure.net/"
	},
	"resourceGroup": "cloud-shell-storage-westeurope",
	"systemData": {
		"createdAt": "2024-06-12T15:48:59.183000+00:00",
		"createdBy": "",
		"createdByType": "User",
		"lastModifiedAt": "2024-06-12T15:48:59.183000+00:00",
		"lastModifiedBy": "",
		"lastModifiedByType": "User"
	},
	"tags": {},
	"type": "Microsoft.KeyVault/vaults"
}

03 Run keyvault set-policy command (Windows/macOS/Linux) with the object ID (i.e. "properties.accessPolicies.objectId") and the name of the newly created key vault as the identifier parameters to assign the right permissions to the selected vault:

az keyvault set-policy
  --name cc-project5-vault
  --object-id abcdabcd-abcd-abcd-abcd-abcdabcdabcd
  --key-permissions create get recover unwrapKey wrapKey
  --query 'properties.accessPolicies'

04 The command output should return the modified key vault configuration information:

[
	{
		"applicationId": null,
		"objectId": "abcdabcd-abcd-abcd-abcd-abcdabcdabcd",
		"permissions": {
			"certificates": [
				"all"
			],
			"keys": [
				"recover",
				"unwrapKey",
				"get",
				"create",
				"wrapKey"
			],
			"secrets": [
				"all"
			],
			"storage": [
				"all"
			]
		},
		"tenantId": "abcd1234-abcd-1234-abcd-1234abcd1234"
	}
]

05 Run keyvault key create command (Windows/macOS/Linux) to create the Customer-Managed Key (CMK) necessary to encrypt data for your Azure Machine Learning workspace:

az keyvault key create
  --name cc-ml-workspace-key
  --vault-name cc-project5-vault
  --kty RSA
  --size 2048
  --ops decrypt encrypt sign unwrapKey verify wrapKey
  --protection software
  --disabled false
  --query 'key.kid'

06 The command output should return the URL of the new Customer-Managed Key:

"https://cc-project5-vault.vault.azure.net/keys/cc-ml-workspace-key/abcd1234abcd1234abcd1234abcd1234"

07 Define the configuration file for your new Azure Machine Learning workspace. Create a new configuration file (YAML format), name the file workspace-config.yml, and paste the following content (replace the configuration parameter values such as key vault ID and key URL with your own information):

$schema: https://azuremlschemas.azureedge.net/latest/workspace.schema.json
name: tm-new-project5-workspace
location: westeurope
display_name: tm-new-project5-workspace
customer_managed_key:
	key_vault: /subscriptions/abcd1234-abcd-1234-abcd-1234abcd1234/resourceGroups/cloud-shell-storage-westeurope/providers/Microsoft.KeyVault/vaults/cc-project5-vault
	key_uri: https://cc-project5-vault.vault.azure.net/keys/cc-ml-workspace-key/abcd1234abcd1234abcd1234abcd1234

08 Run ml workspace create command (Windows/macOS/Linux) with the configuration file defined at the previous step (i.e. workspace-config.yml), to create your new, encrypted Azure Machine Learning workspace:

az ml workspace create
  --resource-group cloud-shell-storage-westeurope
  --file workspace-config.yml

09 The command output should return the new workspace configuration information:

{
	"customer_managed_key": {
		"cosmosdb_id": "",
		"key_uri": "https://cc-project5-vault.vault.azure.net/keys/cc-ml-workspace-key/abcd1234abcd1234abcd1234abcd1234",
		"key_vault": "/subscriptions/abcd1234-abcd-1234-abcd-1234abcd1234/resourceGroups/cloud-shell-storage-westeurope/providers/Microsoft.KeyVault/vaults/cc-project5-vault",
		"search_id": "",
		"storage_id": ""
	},
	"description": "tm-new-project5-workspace",
	"discovery_url": "https://westeurope.api.azureml.ms/discovery",
	"display_name": "tm-new-project5-workspace",
	"enable_data_isolation": false,
	"hbi_workspace": true,
	"id": "/subscriptions/abcd1234-abcd-1234-abcd-1234abcd1234/resourceGroups/cloud-shell-storage-westeurope/providers/Microsoft.MachineLearningServices/workspaces/tm-new-project5-workspace",
	"identity": {
		"principal_id": "abcd1234-abcd-1234-abcd-1234abcd1234",
		"tenant_id": "abcd1234-abcd-1234-abcd-1234abcd1234",
		"type": "system_assigned"
	},
	"key_vault": "/subscriptions/abcd1234-abcd-1234-abcd-1234abcd1234/resourceGroups/cloud-shell-storage-westeurope/providers/Microsoft.Keyvault/vaults/1234abcd12341234abcd1234",
	"location": "westeurope",
	"managed_network": {
		"isolation_mode": "disabled",
		"outbound_rules": []
	},
	"mlflow_tracking_uri": "azureml://westeurope.api.azureml.ms/mlflow/v1.0/subscriptions/abcd1234-abcd-1234-abcd-1234abcd1234/resourceGroups/cloud-shell-storage-westeurope/providers/Microsoft.MachineLearningServices/workspaces/tm-new-project5-workspace",
	"name": "tm-new-project5-workspace",
	"public_network_access": "Enabled",
	"resourceGroup": "cloud-shell-storage-westeurope",
	"resource_group": "cloud-shell-storage-westeurope",
	"serverless_compute": {
		"no_public_ip": false
	},
	"storage_account": "/subscriptions/abcd1234-abcd-1234-abcd-1234abcd1234/resourceGroups/cloud-shell-storage-westeurope/providers/Microsoft.Storage/storageAccounts/1234abcd12341234abcd1234",
	"tags": {
		"AttachAppInsightsToWorkspace": "false",
		"createdByToolkit": "cli-v2-1.16.1"
	}
}

10 Repeat steps no. 7 - 9 for each workspace that you want to re-create, available in the selected Azure subscription.

11 Repeat steps no. 1 – 10 for each subscription available in your Microsoft Azure cloud account.

References

Azure Command Line Interface (CLI) Documentation
az ml workspace list
az ml workspace show
az ml workspace create
az keyvault create
az keyvault set-policy
az keyvault key create

Publication date Jun 19, 2024

Audit

Using Azure Console

Using Azure CLI

Remediation / Resolution

Using Azure Console

Using Azure CLI

References

Related MachineLearning rules