OpenSearch General Purpose SSD

Risk Level: Medium (should be achieved)

Rule ID: ES-001

Ensure that your Amazon OpenSearch cluster data nodes are using General Purpose SSD volumes instead of Provisioned IOPS SSD volumes for cost-effective storage that fits a broad range of workloads. Unless you are running mission-critical applications that require more than 10K IOPS per data node, it is highly recommended to convert your Provisioned IOPS data nodes to General Purpose nodes in order to lower the cost of your AWS bill while maintaining the same I/O performance.

This rule can help you work with the AWS Well-Architected Framework.

This rule resolution is part of the Conformity Security & Compliance tool for AWS.

Cost
optimisation

Using General Purpose (GP) SSD storage instead of Provisioned IOPS (PIOPS) SSD storage for your Amazon OpenSearch cluster data nodes represents a good strategy for cutting down on AWS cloud costs because for GP SSDs you only pay for the storage compared to PIOPS SSDs where you pay for both storage and IOPS. Converting existing PIOPS-based data nodes to GP nodes is possible by configuring larger storage which gives higher baseline performance of IOPS for a lower cost.

Audit

To determine the storage type configured for your OpenSearch cluster data nodes, perform the following operations:

Using AWS Console

01 Sign in to the AWS Management Console.

02 Navigate to Amazon OpenSearch console at https://console.aws.amazon.com/esv3/.

03 In the main navigation panel, under Dashboard, select Domains.

04 Click on the name (link) of the OpenSearch cluster that you want to examine.

05 Select the Cluster configuration tab and check the EBS volume type attribute value available in the Data nodes section. If the EBS volume type is set to Provisioned IOPS (SSD), the storage type configured for your Amazon OpenSearch cluster data nodes is Provisioned IOPS SSD, therefore the verified data nodes are not optimized with respect to cost.

06 Repeat steps no. 4 and 5 for each Amazon OpenSearch cluster available within the current AWS region.

07 Change the AWS cloud region from the navigation bar and repeat the Audit process for other regions.

Using AWS CLI

01 Run list-domain-names command (OSX/Linux/UNIX) to list the name of each Amazon OpenSearch cluster (domain) available in the selected AWS region:

aws es list-domain-names
  --region us-east-1
  --query 'DomainNames[*].DomainName'

02 The command output should return the identifier (name) of each OpenSearch domain provisioned in the selected region:

[
    "trendmicro",
    "cloudconformity"
]

03 Run describe-elasticsearch-domain command (OSX/Linux/UNIX) using the name of the Amazon OpenSearch cluster that you want to examine as the identifier parameter and custom query filters to describe the storage type of the data nodes provisioned for the selected cluster:

aws es describe-elasticsearch-domain
  --region us-east-1
  --domain-name trendmicro
  --query 'DomainStatus.EBSOptions.VolumeType'

04 The command output should return the storage type used for the cluster data nodes:

"io1"

If the describe-elasticsearch-domain command output returns "io1", as shown in the example above, the storage type configured for your Amazon OpenSearch cluster data nodes is Provisioned IOPS SSD, therefore the verified data nodes are not optimized with respect to cost.

05 Repeat steps no. 3 and 4 for each Amazon OpenSearch cluster available in the selected AWS region.

06 Change the AWS cloud region by updating the --region command parameter value and repeat the Audit process for other regions.

Remediation / Resolution

To convert your Provisioned IOPS SSD cluster data nodes to General Purpose SSD data nodes, perform the following operations:

Using AWS CloudFormation

01 CloudFormation template (JSON):

{
	"AWSTemplateFormatVersion": "2010-09-09",
	"Description": "Configure General Purpose SSD (GP2) Data Nodes",
	"Resources": {
		"OpenSearchDomain": {
			"Type":"AWS::OpenSearchService::Domain",
			"Properties": {
				"DomainName": "cc-opensearch-domain",
				"EngineVersion": "OpenSearch_1.1",
				"ClusterConfig": {
					"InstanceType": "t3.small.search",
					"InstanceCount": "2"
				},
				"EBSOptions": {
					"EBSEnabled": true,
					"VolumeType": "gp2",
					"VolumeSize": "350"
				},
				"EncryptionAtRestOptions": {
					"Enabled": true
				},
				"AccessPolicies": {
					"Version":"2012-10-17",
					"Statement":[
						{
							"Effect": "Allow",
							"Principal": {
								"AWS": "arn:aws:iam::123456789012:user/cc-opensearch-user"
							},
							"Action":"es:*",
							"Resource": "arn:aws:es:us-east-1:123456789012:domain/cc-opensearch-domain/*"
						}
					]
				}
			}
		}
	}
}

02 CloudFormation template (YAML):

AWSTemplateFormatVersion: '2010-09-09'
	Description: Configure General Purpose SSD (GP2) Data Nodes
	Resources:
		OpenSearchDomain:
		Type: AWS::OpenSearchService::Domain
		Properties:
			DomainName: cc-opensearch-domain
			EngineVersion: OpenSearch_1.1
			ClusterConfig:
			InstanceType: t3.small.search
			InstanceCount: '2'
			EBSOptions:
			EBSEnabled: true
			VolumeType: gp2
			VolumeSize: '350'
			EncryptionAtRestOptions:
			Enabled: true
			AccessPolicies:
			Version: '2012-10-17'
			Statement:
				- Effect: Allow
				Principal:
					AWS: arn:aws:iam::123456789012:user/cc-opensearch-user
				Action: es:*
				Resource: arn:aws:es:us-east-1:123456789012:domain/cc-opensearch-domain/*

Using Terraform (AWS Provider)

01 Terraform configuration file (.tf):

terraform {
	required_providers {
		aws = {
			source  = "hashicorp/aws"
			version = "~> 4.0"
		}
	}

	required_version = ">= 0.14.9"
}

provider "aws" {
	region  = "us-east-1"
}

resource "aws_opensearch_domain" "opensearch-domain" {
	domain_name = "cc-opensearch-domain"
	engine_version = "OpenSearch_1.1"

	cluster_config {
		instance_type = "t3.small.search"
		instance_count = 1
	}

	ebs_options {
		# Configure General Purpose SSD (GP2) Data Nodes
		ebs_enabled = true
		volume_type = "gp2"
		volume_size = 350
	}

	encrypt_at_rest {
		enabled = true
	}

	access_policies = <<POLICY
	{
		"Version": "2012-10-17",
		"Statement":[
			{
				"Effect": "Allow",
				"Principal": {
					"AWS": "arn:aws:iam::123456789012:user/cc-opensearch-user"
				},
				"Action":"es:*",
				"Resource": "arn:aws:es:us-east-1:123456789012:domain/cc-opensearch-domain/*"
			}
		]
	}
	POLICY

}

Using AWS Console

01 Sign in to the AWS Management Console.

02 Navigate to Amazon OpenSearch console at https://console.aws.amazon.com/esv3/.

03 In the main navigation panel, under Dashboard, select Domains.

04 Select the OpenSearch cluster that you want to reconfigure, choose Actions from the console top menu, and select Edit cluster configuration.

05 In the Data nodes section, perform the following actions:

For EBS volume type, select General Purpose (SSD) to convert your Provisioned IOPS SSD data nodes to General Purpose SSD nodes.
Increase the storage size of the data nodes available in the EBS storage size per node box to match the IOPS number of the original nodes.
Choose Save changes to apply the configuration changes.

06 Repeat steps no. 3 – 5 to change the storage type for other Amazon OpenSearch clusters available within the current AWS region.

07 Change the AWS cloud region from the navigation bar and repeat the Remediation process for other regions.

Using AWS CLI

01 Run update-elasticsearch-domain-config command (OSX/Linux/UNIX) using the name of the Amazon OpenSearch cluster that you want to reconfigure as the identifier parameter to convert your Provisioned IOPS SSD cluster data nodes to General Purpose SSD data nodes:

aws es update-elasticsearch-domain-config
  --domain-name trendmicro
  --region us-east-1
  --ebs-options EBSEnabled=true,VolumeType="gp2",VolumeSize=350

02 The command output should return the configuration information available for the reconfigured cluster:

{
    "DomainConfig": {
        "ElasticsearchVersion": {
            "Options": "7.9",
            "Status": {
                "CreationDate": "2021-12-21T14:44:37.462000+00:00",
                "UpdateDate": "2021-12-21T14:57:39.078000+00:00",
                "UpdateVersion": 5,
                "State": "Active",
                "PendingDeletion": false
            }
        },
        "ElasticsearchClusterConfig": {
            "Options": {
                "InstanceType": "m4.large.elasticsearch",
                "InstanceCount": 2,
                "DedicatedMasterEnabled": false,
                "ZoneAwarenessEnabled": true,
                "ZoneAwarenessConfig": {
                    "AvailabilityZoneCount": 2
                },
                "WarmEnabled": false,
                "ColdStorageOptions": {
                    "Enabled": false
                }
            },
        "EBSOptions": {
            "Options": {
                "EBSEnabled": true,
                "VolumeType": "gp2",
                "VolumeSize": 350
            },
            "Status": {
                "CreationDate": "2021-12-21T14:44:37.462000+00:00",
                "UpdateDate": "2021-12-21T14:57:39.078000+00:00",
                "UpdateVersion": 5,
                "State": "Active",
                "PendingDeletion": false
            }
        },

        ...

        "SnapshotOptions": {
            "Options": {
                "AutomatedSnapshotStartHour": 0
            },
            "Status": {
                "CreationDate": "2021-12-21T14:44:37.462000+00:00",
                "UpdateDate": "2021-12-21T14:57:39.078000+00:00",
                "UpdateVersion": 5,
                "State": "Active",
                "PendingDeletion": false
            }
        },
        "VPCOptions": {
            "Options": {},
            "Status": {
                "CreationDate": "2021-12-21T22:17:52.963000+00:00",
                "UpdateDate": "2021-12-21T22:17:52.963000+00:00",
                "UpdateVersion": 17,
                "State": "Active",
                "PendingDeletion": false
            }
        },
        "CognitoOptions": {
            "Options": {
                "Enabled": false
            },
            "Status": {
                "CreationDate": "2021-12-21T22:17:52.963000+00:00",
                "UpdateDate": "2021-12-21T22:17:52.963000+00:00",
                "UpdateVersion": 17,
                "State": "Active",
                "PendingDeletion": false
            }
        }
    }
}

03 Repeat steps no. 1 and 2 to change the storage type for other Amazon OpenSearch clusters available in the selected AWS region.

04 Change the AWS cloud region by updating the --region command parameter value and repeat the Remediation process for other regions.

References

AWS Command Line Interface (CLI) Documentation
es
list-domain-names
describe-elasticsearch-domain
update-elasticsearch-domain-config

CloudFormation Documentation
Amazon OpenSearch Service (successor to Amazon Elasticsearch Service) resource type reference

Terraform Documentation
AWS Provider

Publication date Dec 3, 2016

Audit

Using AWS Console

Using AWS CLI

Remediation / Resolution

Using AWS CloudFormation

Using Terraform (AWS Provider)

Using AWS Console

Using AWS CLI

References

Related Elasticsearch rules