CloudFormation Stack Failed Status

Risk Level: Medium (should be achieved)

Rule ID: CFM-004

Ensure that none of your Amazon CloudFormation stacks remain in "Failed" mode for more than 6 hours. A stack is on "Failed" mode when its status is set to "CREATE_FAILED" - unsuccessful creation of the stack, "DELETE_FAILED" - unsuccessful deletion of the stack, "ROLLBACK_FAILED" - unsuccessful removal of the stack after the creation process failed, or "UPDATE_ROLLBACK_FAILED" - unsuccessful return of the stack to a previous working state after a failed update.

This rule can help you with the following compliance standards:

APRA
MAS

For further details on compliance standards supported by Conformity, see here.

This rule can help you work with the AWS Well-Architected Framework.

This rule resolution is part of the Conformity Security & Compliance tool for AWS.

Operational
excellence

Any failed Amazon CloudFormation stack that is not fixed on time can lead to application downtime, security issues, or unexpected costs on your AWS bill. For example, the unsuccessful deletion (i.e. "DELETE_FAILED") of one or more stacks can accrue charges for the unused AWS cloud resources provisioned by the stack.

Audit

To determine if there are any failed Amazon CloudFormation stacks available within your AWS account, perform the following operations:

Using AWS Console

01 Sign in to the AWS Management Console.

02 Navigate to Amazon CloudFormation console at https://console.aws.amazon.com/cloudformation/.

03 In the Stacks section, select Failed from the filter dropdown menu to list only the failed Amazon CloudFormation stacks available in the current AWS region.

04 Click on the name (link) of the failed stack that you want to examine.

05 Select the Events tab and check the time when the event occurred, listed in the Timestamp column. If the FAILED mode was triggered more than 6 hours ago, the selected Amazon CloudFormation stack is considered unsuccessful and can be removed from your AWS account.

06 Repeat step no. 4 and 5 for each failed Amazon CloudFormation stack available within the current AWS region.

07 Change the AWS cloud region from the navigation bar and repeat the Audit process for other regions.

Using AWS CLI

01 Run list-stacks command (OSX/Linux/UNIX) to list the name of each Amazon CloudFormation stack available in FAILED mode, within the selected AWS region:

aws cloudformation list-stacks
  --region us-east-1
  --stack-status-filter "CREATE_FAILED" "DELETE_FAILED" "ROLLBACK_FAILED" "UPDATE_FAILED" "UPDATE_ROLLBACK_FAILED" "IMPORT_ROLLBACK_FAILED"
  --output table
  --query 'StackSummaries[*].StackName'

02 The command output should return the name(s) of the failed CloudFormation stack(s):

-----------------------------
|         ListStacks        |
+---------------------------+
| cc-project5-test-stack    |
| cc-demo-application-stack |
+---------------------------+

03 Run describe-stacks command (OSX/Linux/UNIX) using the name of the failed CloudFormation stack that you want to examine as the identifier parameter and custom query filters to describe the events triggered for the selected stack in reverse chronological order:

aws cloudformation describe-stack-events
  --region us-east-1
  --stack-name cc-project5-test-stack
  --query 'StackEvents'

04 The command output should return the requested event information:

[
    {
        "StackId": "arn:aws:cloudformation:us-east-1:123456789012:stack/cc-project5-test-stack/abcdabcd-1234-abcd-1234-abcd1234abcd",
        "EventId": "WebServerSecurityGroup-CREATE_FAILED-2022-01-13T17:11:23.981Z",
        "StackName": "cc-project5-test-stack",
        "LogicalResourceId": "WebServerSecurityGroup",
        "PhysicalResourceId": "cc-project5-test-stack-WebServerSecurityGroup-ABCDABCDABCDABCD",
        "ResourceType": "AWS::EC2::SecurityGroup",
        "Timestamp": "2022-01-13T17:11:23.981000+00:00",
        "ResourceStatus": "CREATE_FAILED",
        "ResourceProperties": "{\"GroupDescription\":\"Enable HTTP access via port 80\",\"SecurityGroupIngress\":[{\"CidrIp\":\"0.0.0.0/0\",\"FromPort\":\"80\",\"ToPort\":\"80\",\"IpProtocol\":\"tcp\"},{\"CidrIp\":\"0.0.0.0/0\",\"FromPort\":\"22\",\"ToPort\":\"22\",\"IpProtocol\":\"tcp\"}]}",
        "ClientRequestToken": "Console-CreateStack-abcdabcd-1234-abcd-1234-abcd1234abcd"
    },
    {
        "StackId": "arn:aws:cloudformation:us-east-1:123456789012:stack/cc-project5-test-stack/abcdabcd-1234-abcd-1234-abcd1234abcd",
        "EventId": "WebServerSecurityGroup-CREATE_IN_PROGRESS-2022-01-13T17:11:17.471Z",
        "StackName": "cc-project5-test-stack",
        "LogicalResourceId": "WebServerSecurityGroup",
        "PhysicalResourceId": "",
        "ResourceType": "AWS::EC2::SecurityGroup",
        "Timestamp": "2022-01-13T17:11:17.471000+00:00",
        "ResourceStatus": "CREATE_IN_PROGRESS",
        "ResourceProperties": "{\"GroupDescription\":\"Enable HTTP access via port 80\",\"SecurityGroupIngress\":[{\"CidrIp\":\"0.0.0.0/0\",\"FromPort\":\"80\",\"ToPort\":\"80\",\"IpProtocol\":\"tcp\"},{\"CidrIp\":\"0.0.0.0/0\",\"FromPort\":\"22\",\"ToPort\":\"22\",\"IpProtocol\":\"tcp\"}]}",
        "ClientRequestToken": "Console-CreateStack-abcdabcd-1234-abcd-1234-abcd1234abcd"
    },
    {
        "StackId": "arn:aws:cloudformation:us-east-1:123456789012:stack/cc-project5-test-stack/abcdabcd-1234-abcd-1234-abcd1234abcd",
        "EventId": "abcdabcd-1234-abcd-1234-abcd1234abcd",
        "StackName": "cc-project5-test-stack",
        "LogicalResourceId": "cc-project5-test-stack",
        "PhysicalResourceId": "arn:aws:cloudformation:us-east-1:123456789012:stack/cc-project5-test-stack/abcdabcd-1234-abcd-1234-abcd1234abcd",
        "ResourceType": "AWS::CloudFormation::Stack",
        "Timestamp": "2022-01-13T17:11:10.497000+00:00",
        "ResourceStatus": "CREATE_IN_PROGRESS",
        "ResourceStatusReason": "User Initiated",
        "ClientRequestToken": "Console-CreateStack-abcdabcd-1234-abcd-1234-abcd1234abcd"
    }
]

Check the "Timestamp" property value listed for the failed stack event (highlighted) to determine when the event occurred and switched on the FAILED mode. Based on the timestamp returned, if the FAILED mode was triggered more than 6 hours ago, the selected Amazon CloudFormation stack is considered unsuccessful and can be removed from your AWS account.

05 Repeat steps no. 3 and 4 for each failed Amazon CloudFormation stack available in the selected AWS region.

06 Change the AWS cloud region by updating the --region command parameter value and repeat the Audit process for other regions.

Remediation / Resolution

To remove Amazon CloudFormation stacks available in FAILED mode for more than 6 hours, perform the following operations:

Using Terraform (AWS Provider)

01 Terraform configuration file (.tf):

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 3.27"
    }
  }

  required_version = ">= 0.14.9"
}

provider "aws" {
  profile = "default"
  region  = "us-east-1"
}

resource "aws_cloudformation_stack" "prod-vpc-network" {
  name = "cc-prod-vpc-network"
}

02 Terraform commands:

terraform destroy -target aws_cloudformation_stack.prod-vpc-network

Using AWS Console

01 Sign in to the AWS Management Console.

02 Navigate to Amazon CloudFormation console at https://console.aws.amazon.com/cloudformation/.

03 Select the failed Amazon CloudFormation stack that you want to delete.

04 Choose Delete from the console top menu to initiate the stack removal.

05 In the Delete ? confirmation box, review the stack details, then choose Delete stack to remove the selected Amazon CloudFormation stack from your AWS account. The stack status should change now to DELETE_IN_PROGRESS. Once the selected stack and all its resources have been successfully deleted, Amazon CloudFormation will remove the stack entry from the Stacks listing page.

06 Repeat steps no. 3 – 5 to remove each failed Amazon CloudFormation stack available within the current AWS region.

07 Change the AWS cloud region from the navigation bar and repeat the Remediation process for other regions.

Using AWS CLI

01 Run delete-stack command (OSX/Linux/UNIX) using the name of the failed Amazon CloudFormation stack that you want to delete, to remove the selected stack and its resources from your AWS account (the command does not produce an output):

aws cloudformation delete-stack
  --region us-east-1
  --stack-name cc-project5-test-stack

02 Repeat step no. 1 to remove each failed Amazon CloudFormation stack available in the selected AWS region.

03 Change the AWS cloud region by updating the --region command parameter value and repeat the Remediation process for other regions.

References

AWS Documentation
AWS CloudFormation Concepts
Describing and listing your stacks
Troubleshooting CloudFormation
Deleting a stack on the AWS CloudFormation console

AWS Command Line Interface (CLI) Documentation
cloudformation
list-stacks
describe-stacks
update-termination-protection
delete-stack

Terraform Documentation
AWS Provider

Publication date Feb 6, 2017

Audit

Using AWS Console

Using AWS CLI

Remediation / Resolution

Using Terraform (AWS Provider)

Using AWS Console

Using AWS CLI

References

Related CloudFormation rules