- Knowledge Base
- Google Cloud Platform
- GCP Eventarc Service
- Configure Dead Lettering for Topics Associated with Eventarc Triggers
Ensure that Pub/Sub topics associated with Eventarc triggers have a Dead-Letter Topic (DLT) configured to handle undeliverable messages. To achieve this, configure your Pub/Sub subscriptions with a maximum number of delivery attempts. If a message cannot be delivered, it will be sent to the designated Dead-Letter Topic (DLT).
excellence
When a message can't be acknowledged, Google Cloud Pub/Sub can forward the message to a Dead-Letter Topic (DLT) that stores the message for later access. Configuring Dead-Letter Topics (DLTs) for your Pub/Sub-triggered functions can make your serverless applications more resilient and durable by capturing and storing messages that can't be delivered due to client errors or server errors for further analysis or reprocessing.
Audit
To determine if dead lettering is enabled for the associated Pub/Sub topics, perform the following operations:
Using GCP Console
01 Sign in to the Google Cloud Management Console.
02 Select the Google Cloud Platform (GCP) project that you want to examine from the console top navigation bar.
03 Navigate to Eventarc console available at https://console.cloud.google.com/eventarc/.
04 In the left navigation panel, under Standard, choose Triggers to access the list of Eventarc triggers available for the selected GCP project.
05 Click on the name (link) of the Eventarc trigger that you want to examine, listed in the Name column.
06 In the Details section, click on the ID (link) of the Pub/Sub topic associated with the selected Eventarc trigger, listed next to Topic.
07 Select the SUBSCRIPTION tab and click on the identifier (ID) of the subscription that you want to examine, listed in the Subcription ID column.
08 Choose the DETAILS tab and check the Dead lettering feature status. If the Dead lettering status is set to Disabled, the selected Pub/Sub subscription does not have a Dead-Letter Topic (DLT) configured to capture undeliverable messages.
09 Repeat steps no. 7 and 8 for each Pub/Sub subscription created for the associated topic. If none of these subscriptions has a Dead-Letter Topic (DLT) set up to handle undeliverable messages, dead lettering is not configured for the Pub/Sub topic associated with your Eventarc trigger.
10 Repeat step no. 5 - 9 for each Eventarc trigger available within the selected GCP project.
11 Repeat steps no. 2 - 10 for each GCP project deployed within your Google Cloud account.
Using GCP CLI
01 Run projects list command (Windows/macOS/Linux) with custom output filters to list the ID of each GCP project available in your Google Cloud account:
gcloud projects list --format="table(projectId)"
02 The command output should return the requested GCP project IDS:
PROJECT_ID cc-web-project-123123 cc-ai-project-112233 cc-dev-project-112233
03 Run eventarc triggers list command (Windows/macOS/Linux) with the ID of the GCP project that you want to examine as the identifier parameter and custom output filters to describe the ID of each Eventarc trigger created for the selected project:
gcloud eventarc triggers list --project=cc-web-project-123123 --format="default(name)"
04 The command request should return the requested Eventarc trigger IDs (i.e., fully qualified identifiers):
--- name: projects/cc-web-project-123123/locations/us-central1/triggers/cc-project5-trigger --- name: projects/cc-web-project-123123/locations/us-central1/triggers/cc-cloud-ai-trigger
05 Run eventarc triggers describe command (Windows/macOS/Linux) with the ID of the Eventarc trigger that you want to examine as the identifier parameter and custom output filters to describe the identifier (ID) of the Pub/Sub topic associated with the selected trigger:
gcloud eventarc triggers describe "projects/cc-web-project-123123/locations/us-central1/triggers/cc-project5-trigger" --format="value(transport.pubsub.topic)"
06 The command output should return the ID of the requested Pub/Sub topic:
projects/cc-web-project-123123/topics/eventarc-us-central1-cc-project5-trigger-170
07 Run pubsub topics list-subscriptions command (Windows/macOS/Linux) to describe the fully qualified identifier of each subscription created for the Pub/Sub topic associated with your trigger (returned at the previous step):
gcloud pubsub topics list-subscriptions projects/cc-web-project-123123/topics/eventarc-us-central1-cc-project5-trigger-170 --format="json"
08 The command output should return the requested subscription identifiers:
[ "projects/cc-web-project-123123/subscriptions/eventarc-us-central1-cc-project5-eventarc-trigger-sub-165", "projects/cc-web-project-123123/subscriptions/eventarc-us-central1-cc-project5-eventarc-trigger-sub-166" ]
09 Run pubsub subscriptions describe command (Windows/macOS/Linux) with custom output filters to describe the identifier (ID) of the Dead-Letter Topic (DLT) configured for the specified Pub/Sub subscription:
gcloud pubsub subscriptions describe projects/cc-web-project-123123/subscriptions/eventarc-us-central1-cc-project5-eventarc-trigger-sub-165 --format="json(deadLetterPolicy.deadLetterTopic)"
10 The command output should return the requested DLT identifier:
null
If the pubsub subscriptions describe command output returns null, as shown in the example above, the selected Pub/Sub subscription does not have a Dead-Letter Topic (DLT) configured to capture undeliverable messages.
11 Repeat steps no. 9 and 10 for each Pub/Sub subscription created for the associated topic. If none of these subscriptions has a Dead-Letter Topic (DLT) set up to handle undeliverable messages, dead lettering is not configured for the Pub/Sub topic associated with your Eventarc trigger.
12 Repeat steps no. 5 - 11 for each Eventarc trigger available within the selected GCP project.
13 Repeat steps no. 3 – 12 for each GCP project deployed in your Google Cloud account.
Remediation / Resolution
To ensure that your Pub/Sub topics associated with Eventarc triggers have a Dead-Letter Topic (DLT) configured to handle undeliverable messages, perform the following operations:
Using GCP Console
01 Sign in to the Google Cloud Management Console.
02 Select the Google Cloud Platform (GCP) project that you want to access from the console top navigation bar.
03 It is strongly recommended to utilize a separate Pub/Sub topic for your Dead-Letter Topic (DLT). Navigate to Pub/Sub console available at https://console.cloud.google.com/cloudpubsub, choose CREATE TOPIC, and perform the following actions to set up your Dead-Letter Topic:
- Provide a unique identifier for the new topic in the Topic ID box.
- Your Dead-Letter Topic (DLT) should have at least one subscription so that dead-lettered messages will not be lost. Check the Add a default subscription setting checkbox to create a subscription for your new DLT.
- For Encryption, choose Cloud KMS key, and select your own Customer-Managed Encryption Key (CMEK) from the Select a Cloud KMS key dropdown list.
- Choose CREATE to deploy your new Dead-Letter Topic (DLT).
04 Navigate to Eventarc console available at https://console.cloud.google.com/eventarc/.
05 In the left navigation panel, under Standard, choose Triggers to access the list of Eventarc triggers available for the selected GCP project.
06 Click on the name (link) of the Eventarc trigger that you want to examine, listed in the Name column.
07 In the Details section, click on the ID (link) of the Pub/Sub topic associated with the selected Eventarc trigger, listed next to Topic.
08 Select the SUBSCRIPTION tab and click on the identifier (ID) of the subscription that you want to configure, listed in the Subcription ID column.
09 Choose EDIT and perform the following actions to enable dead lettering for the selected subscription:
- In the Dead lettering section, check the Enable dead lettering checkbox, and choose the Dead-Letter Topic (DLT) created earlier in the Remediation section from the Select a dead letter topic dropdown list.
- (Optional) For Maximum delivery attempts, specify an integer between 5 and 100, based on your application needs.
- Choose UPDATE to apply the configuration changes.
10 Select the DEAD LETTERING tab, and click on the GRANT PUBLISHER ROLE and GRANT SUBSCRIBER ROLE links to assign the Publisher and Subscriber roles. The Pub/Sub service account for the selected GCP project needs the Publisher role to publish dead-lettered messages to the specified topic and the Subscriber role to forward messages from this subscription to the Dead-Letter Topic (DLT). The selected subscription can dead letter messages to the new DLT once it exceeds the specified delivery attempt count.
11 Repeat steps no. 6 - 10 for each Eventarc trigger that you want to configure, available within the selected GCP project.
12 Repeat steps no. 2 – 11 for each project deployed in your Google Cloud account.
Using GCP CLI
01 The best practice is to use a dedicated Pub/Sub topic for your Dead-Letter Topic (DLT). Run pubsub topics create command (Windows/macOS/Linux) to create the Pub/Sub Dead-Letter Topic (DLT) that will capture undeliverable messages:
gcloud pubsub topics create cc-dead-letter-topic --project cc-web-project-123123
02 The command output should return the full resource identifier of the newly created topic:
Created topic [projects/cc-web-project-123123/topics/cc-dead-letter-topic].
03 The new DLT should have at least one subscription so that dead-lettered messages will not be lost. Run pubsub subscriptions create command (Windows/macOS/Linux) to create the required subscription for your new Dead-Letter Topic (DLT):
gcloud pubsub subscriptions create cc-dead-letter-subscription --topic=cc-dead-letter-topic --topic-project=cc-web-project-123123
04 The command output should return the resource ID of the new Pub/Sub subscription:
Created subscription [projects/cc-web-project-123123/subscriptions/cc-dead-letter-subscription].
05 Run pubsub subscriptions update command (Windows/macOS/Linux) to enable dead lettering for the subscription associated with your Eventarc trigger:
gcloud pubsub subscriptions update projects/cc-web-project-123123/subscriptions/eventarc-us-central1-cc-project5-eventarc-trigger-sub-165 --dead-letter-topic=cc-dead-letter-topic --max-delivery-attempts=10 --dead-letter-topic-project=cc-web-project-123123
06 The command output should return the URL of the configured Pub/Sub subscription:
Updated subscription [projects/cc-web-project-123123/subscriptions/eventarc-us-central1-cc-project5-eventarc-trigger-sub-165].
07 The Pub/Sub service account for the selected Google Cloud project needs the Publisher role to publish dead-lettered messages to the newly created Dead-Letter Topic (DLT). To grant Pub/Sub permission to publish messages to the specified topic, run pubsub topics add-iam-policy-binding command (Windows/macOS/Linux). Replace \<project-number\> with your own GCP project number:
gcloud pubsub topics add-iam-policy-binding cc-dead-letter-topic --member="serviceAccount:service-<project-number>@gcp-sa-pubsub.iam.gserviceaccount.com" --role="roles/pubsub.publisher"
08 The command output should return the IAM policy binding information:
Updated IAM policy for topic [cc-dead-letter-topic]. bindings: - members: - serviceAccount:service-<project-number>@gcp-sa-pubsub.iam.gserviceaccount.com role: roles/pubsub.publisher etag: abcdabcdabcd version: 1
09 The Pub/Sub service account for the selected Google Cloud project needs the Subscriber role to forward messages from the configured subscription to the new DLT. To grant Pub/Sub permission to acknowledge forwarded messages and remove them from the specified subscription, run pubsub subscriptions add-iam-policy-binding command (Windows/macOS/Linux). Replace \<project-number\> with your own project number:
gcloud pubsub subscriptions add-iam-policy-binding projects/cc-web-project-123123/subscriptions/eventarc-us-central1-cc-project5-eventarc-trigger-sub-165 --member="serviceAccount:service-<project-number>@gcp-sa-pubsub.iam.gserviceaccount.com" --role="roles/pubsub.subscriber"
10 The command output should return the new IAM policy binding information:
Updated IAM policy for subscription [eventarc-us-central1-cc-project5-eventarc-trigger-sub-165]. bindings: - members: - serviceAccount:service-<project-number>@gcp-sa-pubsub.iam.gserviceaccount.com role: roles/pubsub.subscriber etag: abcdabcdabcd version: 1
11 Repeat steps no. 5 - 10 for each Eventarc trigger that you want to configure, available in the selected project.
12 Repeat steps no. 1 – 11 for each GCP project deployed in your Google Cloud account.
References
- Google Cloud Platform (GCP) Documentation
- Eventarc Standard overview
- Handle message failures
- GCP Command Line Interface (CLI) Documentation
- gcloud projects list
- gcloud eventarc triggers list
- gcloud eventarc triggers describe
- gcloud pubsub subscriptions list
- gcloud pubsub subscriptions describe
- gcloud pubsub subscriptions add-iam-policy-binding
- gcloud pubsub subscriptions create
- gcloud pubsub subscriptions update
- gcloud pubsub topics create
- gcloud pubsub topics add-iam-policy-binding
Related Eventarc rules
- Enable Data Access Audit Logs for Eventarc Resources (Security, reliability, cost-optimisation, performance-efficiency, operational-excellence)
- Configure Dead Lettering for Topics Associated with Eventarc Triggers (Reliability, operational-excellence)
- Use Labels for Resource Management (Security, reliability, performance-efficiency, cost-optimisation)