Use the Conformity Knowledge Base AI to help improve your Cloud Posture

SQS Dead Letter Queue

Trend Cloud One™ – Conformity is a continuous assurance tool that provides peace of mind for your cloud infrastructure, delivering over 1000 automated best practice checks.

Risk Level: Low (generally tolerable level of risk)
Rule ID: SQS-006

Ensure that each Amazon SQS queue is configured to use a Dead-Letter Queue (DLQ) in order to help maintain the queue flow and avoid losing data by detecting and mitigating failures and service disruptions on time. A Dead-Letter Queue is an SQS queue useful for debugging your application or your messaging system, which can isolate messages that can't be processed successfully for later analysis.

This rule can help you with the following compliance standards:

  • NIST4

For further details on compliance standards supported by Conformity, see here.

This rule can help you work with the AWS Well-Architected Framework.

This rule resolution is part of the Conformity Security & Compliance tool for AWS.

Operational
excellence
Reliability

Enabling Dead-Letter Queues (DLQs) for Amazon SQS queues can help you troubleshoot incorrect message transmission operations that can lead to data loss. Use DLQs to decrease the number of unprocessed messages and reduce the possibility of exposing your SQS queues to poison pill messages (i.e. messages that are received but can't be processed).


Audit

To determine if Dead-Letter Queues are configured for all your SQS queues, perform the following operations:

Using AWS Console

01 Sign in to the AWS Management Console.

02 Navigate to Amazon SQS console at https://console.aws.amazon.com/sqs/.

03 In the main navigation panel, under Amazon SQS, choose Queues.

04 Click on the name (link) of the SQS queue that you want to examine.

05 Select the Dead-letter queue tab from the console bottom panel and check for any Dead-Letter Queues (DLQs) configured for the selected SQS queue. If there are no DLQs listed in this section, instead and the following message is displayed: "No dead-letter queue set for this queue.", the selected Amazon SQS queue does not have a Dead-Letter Queue (DLQ) configured to capture undeliverable messages.

06 Repeat steps no. 4 and 5 for each Amazon SQS queue available within the current AWS region.

07 Change the AWS cloud region from the navigation bar and repeat the Audit process for other regions.

Using AWS CLI

01 Run list-queues command (OSX/Linux/UNIX) to list the URL of each Amazon SQS queue available in the selected AWS cloud region:

aws aws sqs list-queues
  --region us-east-1
  --query 'QueueUrls[*]'

02 The command output should return an array with the requested SQS queue URLs:

[
    "https://sqs.us-east-1.amazonaws.com/123456789012/cc-web-app-worker",
    "https://sqs.us-east-1.amazonaws.com/123456789012/cc-mobile-app-queue"
]

03 Run get-queue-attributes command (OSX/Linux/UNIX) using the URL of the SQS queue that you want to examine as the identifier parameter and custom query filters to return the Redrive policy, a string that includes the parameters available for the Dead-Letter Queue functionality of the source queue:

aws sqs get-queue-attributes
  --region us-east-1
  --queue-url https://sqs.us-east-1.amazonaws.com/123456789012/cc-web-app-worker
  --attribute-names RedrivePolicy
  --query 'Attributes.RedrivePolicy'

04 The command output should return the Redrive policy defined for the selected SQS queue:

null

If the get-queue-attributes command output returns null, as shown in the example above, there is no Redrive policy defined, therefore the selected Amazon SQS queue is not associated with a Dead-Letter Queue (DLQ) configured to capture undeliverable messages.

05 Repeat steps no. 3 and 4 for each Amazon SQS queue available in the selected AWS region.

06 Change the AWS cloud region by updating the --region command parameter value and repeat the Audit process for other regions.

Remediation / Resolution

To create and configure a Dead-Letter Queue (DLQ) in order to prevent endless processing of invalid messages for your Amazon SQS queues, perform the following operations:

Using AWS CloudFormation

01 CloudFormation template (JSON):

{
	"AWSTemplateFormatVersion": "2010-09-09",
	"Description": "Create and Configure a Dead-Letter Queue (DLQ) for SQS Queue",
	"Parameters": {
		"SQSQueueName": {
			"Default": "cc-worker-queue",
			"Description": "SQS Worker Queue",
			"Type": "String",
			"MinLength": "1",
			"MaxLength": "63",
			"AllowedPattern": "^[0-9a-zA-Z-/]*$",
			"ConstraintDescription": "Must begin with a letter and must not end with a hyphen or contain two consecutive hyphens."
		}
	},
	"Resources": {
		"SQSDeadLetterQueue": {
			"Type": "AWS::SQS::Queue"
		},
		"SQSSourceQueue": {
			"Type": "AWS::SQS::Queue",
			"Properties": {
				"QueueName": {
					"Ref": "SQSQueueName"
				},
				"RedrivePolicy": {
					"deadLetterTargetArn": {
						"Fn::GetAtt": ["SQSDeadLetterQueue", "Arn"]
					},
					"maxReceiveCount": 10
				}
			}
		}
	}
}

02 CloudFormation template (YAML):

AWSTemplateFormatVersion: '2010-09-09'
	Description: Create and Configure a Dead-Letter Queue (DLQ) for SQS Queue
	Parameters:
		SQSQueueName:
		Default: cc-worker-queue
		Description: SQS Worker Queue
		Type: String
		MinLength: '1'
		MaxLength: '63'
		AllowedPattern: ^[0-9a-zA-Z-/]*$
		ConstraintDescription: Must begin with a letter and must not end with a hyphen
			or contain two consecutive hyphens.
	Resources:
		SQSDeadLetterQueue:
		Type: AWS::SQS::Queue
		SQSSourceQueue:
		Type: AWS::SQS::Queue
		Properties:
			QueueName: !Ref 'SQSQueueName'
			RedrivePolicy:
			deadLetterTargetArn: !GetAtt 'SQSDeadLetterQueue.Arn'
			maxReceiveCount: 10

Using Terraform (AWS Provider)

01 Terraform configuration file (.tf):

terraform {
	required_providers {
		aws = {
			source  = "hashicorp/aws"
			version = "~> 3.27"
		}
	}

	required_version = ">= 0.14.9"
}

provider "aws" {
	profile = "default"
	region  = "us-east-1"
}

# Create Dead-Letter Queue (DLQ) 
resource "aws_sqs_queue" "sqs-queue-deadletter" {
	name = "cc-dead-letter-queue"
}

resource "aws_sqs_queue" "sqs-queue" {
	name                  = "sqs-worker-queue"

	# Configure Dead-Letter Queue (DLQ) 
	redrive_policy = jsonencode({
		deadLetterTargetArn = aws_sqs_queue.sqs-queue-deadletter.arn
		maxReceiveCount     = 10
	})

}

Using AWS Console

01 Sign in to the AWS Management Console.

02 Navigate to Amazon SQS console at https://console.aws.amazon.com/sqs/.

03 In the main navigation panel, under Amazon SQS, choose Queues.

04 To set up a new Dead-Letter Queue (DLQ), choose Create queue.

05 On the Create queue setup page, select Standard for the SQS queue type, enter a unique name for the queue in the Name box, and choose Create queue to create your new Dead-Letter Queue (DLQ).

06 Navigate back to the Queues page and click on the name of the SQS queue that you want to reconfigure.

07 Select the Dead-letter queue tab from the console bottom panel and choose Edit.

08 On the Edit <queue-name> configuration page, select the Dead-letter queue – Optional tab and perform the following actions:

  1. Select Enabled under Set this queue to receive undeliverable messages.
  2. Choose the ARN of the Dead-Letter Queue (DLQ) created at the previous steps from the Choose queue dropdown list.
  3. For Maximum receives, enter the maximum number of times an SQS message can be received before it is sent to the Dead-Letter Queue (DLQ). The value must be between 1 and 1000.
  4. Choose Save to apply the changes. The selected Dead-Letter Queue (DLQ) can now receive unprocessed messages from the source SQS queue.

09 Repeat steps no. 6 – 8 to configure DLQs for other Amazon SQS queues available within the current AWS region.

10 Change the AWS cloud region from the navigation bar and repeat the Remediation process for other AWS regions.

Using AWS CLI

01 Run create-queue command (OSX/Linux/UNIX) to set up the necessary Dead-Letter Queue (DLQ):

aws sqs create-queue
  --region us-east-1
  --queue-name cc-dead-letter-queue

02 The command output should return the complete URL of the new SQS queue:

{
  "QueueUrl": "https://sqs.us-east-1.amazonaws.com/466594415815/cc-dead-letter-queue"
}

03 Define the Redrive policy as shown in the example below and save the policy to a JSON file named cc-redrive-policy.json. The Redrive policy will enable your Amazon SQS queue to send unprocessed messages to the newly created Dead-Letter Queue:

{
  "RedrivePolicy": "{\"deadLetterTargetArn\":\"arn:aws:sqs:us-east-1:123456789012:cc-dead-letter-queue\",\"maxReceiveCount\":\"10\"}"
}

04 Run set-queue-attributes command (OSX/Linux/UNIX) using the URL of the Amazon SQS queue that you want to reconfigure as the identifier parameter and the policy document defined at the previous step (i.e. cc-redrive-policy.json) to implement the Redrive policy and enable the Dead-Letter Queue for the selected SQS queue (the command does not produce an output):

aws sqs set-queue-attributes
  --queue-url https://sqs.us-east-1.amazonaws.com/123456789012/cc-web-app-worker
  --attributes file://cc-redrive-policy.json

05 Repeat steps no. 3 and 4 to configure DLQs for other Amazon SQS queues available in the selected AWS region.

06 Change the AWS cloud region by updating the --region command parameter value and repeat steps no. 1 – 5 to perform the Remediation process for other regions.

References

Publication date Sep 10, 2018