Use the Conformity Knowledge Base AI to help improve your Cloud Posture

Redshift Disk Space Usage

Trend Cloud One™ – Conformity is a continuous assurance tool that provides peace of mind for your cloud infrastructure, delivering over 1000 automated best practice checks.

Risk Level: High (not acceptable risk)
Rule ID: RS-020

Identify Amazon Redshift data warehouse clusters with high disk usage and scale them horizontally to prevent any issues triggered by insufficient disk space and improve their I/O performance. The default threshold value set for Redshift high disk usage is 90% as any value above this could negatively affect cluster stability and performance.
The AWS CloudWatch metric utilized to detect Redshift clusters with high disk space usage is:
PercentageDiskSpaceUsed – the percent of disk space used. (Units: Percent).

This rule can help you work with the AWS Well-Architected Framework.

This rule resolution is part of the Conformity Security & Compliance tool for AWS.

Sustainability
Performance
efficiency

Detecting AWS Redshift clusters with high disk space usage is very important when these clusters are used in production as this can help you take immediate actions and scale out (resize) them in order to respond to data warehousing capacity and performance needs.

Note: You can change the default threshold value (90%) for this rule on Cloud Conformity console and set your own value for high disk space usage to configure the storage limits for your Redshift clusters.


Audit

To identify existing AWS Redshift clusters with high disk space usage, provisioned in your AWS account, perform the following:

Using AWS Console

01 Sign in to AWS Management Console.

02 Navigate to Redshift dashboard at https://console.aws.amazon.com/redshift/.

03 In the left navigation panel, under Redshift Dashboard, click Clusters.

04 Choose the Redshift cluster that you want to examine then click on its identifier/name link, listed in the Cluster column.

05 Click on Performance tab from the dashboard top menu to access the cluster monitoring panel.

06 On the Redshift cluster monitoring panel, set the following parameters:

  • From the Time Range dropdown list, select Last 24 Hours.
  • From the Period list, select 1 Hour.
  • From the Statistic dropdown list, select Maximum.
  • From the Metrics dropdown list, select PercentageDiskSpaceUsed.

Once the monitoring data is loaded, verify the percent of the disk space used currently for the selected cluster. If the amount of disk space used is greater than or equal to 90%, the selected Amazon Redshift cluster has insufficient disk space to perform optimally, therefore scaling the cluster to add more storage space is highly recommended.

07 Repeat steps no. 4 –6 to verify the disk space usage details for other Redshift clusters provisioned in the current region.

08 Change the AWS region from the navigation bar and repeat the audit process for other regions.

Using AWS CLI

01 Run describe-clusters command (OSX/Linux/UNIX) using custom query filters to list the IDs of all AWS Redshift clusters created in the selected region:

aws redshift describe-clusters
	--region us-east-1
	--output table
	--query 'Clusters[*].ClusterIdentifier'

02 The command output should return a table with the requested cluster IDs:

--------------------------
|    DescribeClusters    |
+------------------------+
|  cc-prod-data-cluster  |
|  cc-sandbox-db-cluster |
+------------------------+

03 Run get-metric-statistics command (OSX/Linux/UNIX) to get the statistics recorded by AWS CloudWatch for the PercentageDiskSpaceUsed metric representing the percent of disk space used by the selected Redshift cluster. The following command example returns the percent of disk space used by an AWS Redshift cluster identified by the name cc-prod-data-cluster, usage data captured during a 24 hours time frame, using 1 hour time range as the granularity of the returned datapoints:

aws cloudwatch get-metric-statistics
	--region us-east-1
	--metric-name PercentageDiskSpaceUsed
	--start-time 2017-09-11T11:05:41
	--end-time 2017-09-12T11:05:41
	--period 3600
	--namespace AWS/Redshift
	--statistics Maximum
	--dimensions="Name=ClusterIdentifier,Value=cc-prod-data-cluster"

04 The command output should return the disk usage details requested:

{
    "Datapoints": [
        {
            "Timestamp": "2017-09-11T11:05:41Z",
            "Maximum": 96.2085,
            "Unit": "Percent"
        },
        {
            "Timestamp": "2017-09-11T12:05:41Z",
            "Maximum": 96.2085,
            "Unit": "Percent"
        },
        {
            "Timestamp": "2017-09-11T13:05:41Z",
            "Maximum": 96.2085,
            "Unit": "Percent"
        },

        ...

        {
            "Timestamp": "2017-09-12T09:05:41Z",
            "Maximum": 97.52783333333333333,
            "Unit": "Percent"
        },
        {
            "Timestamp": "2017-09-12T10:05:41Z",
            "Maximum": 97.52783333333333333,
            "Unit": "Percent"
        },
        {
            "Timestamp": "2017-09-12T11:05:41Z",
            "Maximum": 97.52783333333333333,
            "Unit": "Percent"
        }
    ],
    "Label": "PercentageDiskSpaceUsed"
}

If the amount of disk space used is greater than or equal to 90%, the selected AWS Redshift cluster has insufficient disk space to perform optimally.

05 Repeat step no. 3 and 4 to check the disk space usage details for other Redshift clusters provisioned in the current region.

06 Change the AWS region by updating the --region command parameter value and repeat steps no. 1 - 5 to perform the entire audit process for other regions.

Remediation / Resolution

The scale-out (resize) process will just add new nodes to your cluster as is not possible yet to increase the Redshift node(s) disk size. To resize an AWS Redshift cluster in order to increase its storage capacity, perform the following actions:

Note: As example, this section will demonstrate how to scale out a Redshift cluster by resizing it from a single node cluster to a multi-node cluster.

Using AWS Console

01 Sign in to AWS Management Console.

02 Navigate to Redshift dashboard at https://console.aws.amazon.com/redshift/.

03 In the left navigation panel, under Redshift Dashboard, click Clusters.

04 Choose the Redshift cluster that you want to resize (see Audit section part I to identify the right resource), then click on its identifier/name link, listed in the Cluster column.

05 On the selected cluster configuration page, click the Cluster dropdown button from the dashboard top menu and select Resize.

06 Within Resize Cluster configuration window, perform the following:

  1. Select Multi Node from the Cluster type dropdown list to configure the selected cluster into a multi-node cluster.
  2. Increase the number of nodes available within Number of nodes box.
  3. Click Resize to apply the changes and resize the selected Redshift cluster. The Cluster Status should change now to resizing. IMPORTANT: When you start the resize operation, the Redshift service puts the existing cluster into read-only mode until the resize is complete. During this time, you can only run queries that read from the cluster database and you cannot run any queries that write to the database (including read-write queries).

07 Repeat steps no. 4 – 6 to increase the storage capacity for other Amazon Redshift clusters with high disk space usage, provisioned in the current region.

08 Change the AWS region from the navigation bar and repeat the remediation/resolution process for other regions.

Using AWS CLI

01 Run modify-cluster command (OSX/Linux/UNIX) using the name of the cluster that you want to resize as identifier (see Audit section part II to identify the right Redshift resource) to resize (scale-out) the selected Redshift cluster by adding more nodes to its configuration. IMPORTANT: When you execute the modify-cluster command and start the resize operation, the AWS Redshift service puts the existing cluster into read-only mode until the resize is complete. During this time, you can only run queries that read from the cluster database and you cannot run any queries that write to the database (including read-write queries):

aws redshift modify-cluster
	--region us-east-1
	--cluster-identifier cc-prod-data-cluster
	--cluster-type multi-node
	--number-of-nodes 2
	--node-type ds1.xlarge

02 If successful, the command output should return the new configuration metadata for the selected AWS Redshift cluster:

{
    "Cluster": {
        "PubliclyAccessible": true,
        "VpcSecurityGroups": [
            {
                "Status": "active",
                "VpcSecurityGroupId": "sg-04d8d412"
            }
        ],
        "NumberOfNodes": 1,
        "PendingModifiedValues": {
            "NodeType": "ds1.xlarge",
            "ClusterType": "multi-node",
            "NumberOfNodes": 2
        },
        "VpcId": "vpc-3de5f538",
        "ClusterVersion": "1.0",
        "AutomatedSnapshotRetentionPeriod": 7,

        ...

        "ClusterParameterGroups": [
            {
                "ParameterGroupName": "default.redshift-1.0",
                "ParameterApplyStatus": "in-sync"
            }
        ],
        "PreferredMaintenanceWindow": "fri:08:30-fri:09:00",
        "IamRoles": [],
        "AllowVersionUpgrade": true,
        "ClusterCreateTime": "2017-08-10T08:38:15.504Z",
        "ClusterSubnetGroupName": "default",
        "EnhancedVpcRouting": false,
        "ClusterSecurityGroups": [],
        "ClusterIdentifier": "cc-prod-data-cluster",
        "AvailabilityZone": "us-east-1a",
        "NodeType": "ds1.xlarge",
        "Encrypted": false,
        "ClusterStatus": "resizing"
    }
}

03 Repeat step no. 1 and 2 to increase the storage capacity for other Amazon Redshift clusters with high disk space usage, provisioned in the current region.

04 Change the AWS region by updating the --region command parameter value and repeat the entire process for other regions.

References

Publication date Jan 9, 2018