Use the Conformity Knowledge Base AI to help improve your Cloud Posture

Export All Log Entries Using Sinks

Trend Micro Cloud One™ – Conformity is a continuous assurance tool that provides peace of mind for your cloud infrastructure, delivering over 750 automated best practice checks.

Risk Level: Medium (should be achieved)
Rule ID: CloudLogging-009

Ensure there is at least one sink used to export copies of all the log entries available within your Google Cloud Platform (GCP) project. A sink is an object created to hold a log query and a destination. You can export logs by creating one or more log sinks that include a log query and an export destination. As Google Cloud Logging service receives new log entries, they are compared against each sink. If a log entry matches a sink object query, then a copy of the log entry is written to the sink's export destination.

This rule resolution is part of the Conformity Security & Compliance tool for GCP.

Security
Reliability

You can export copies of your logs outside of Google Cloud Logging environment using log sinks. You might want to export your logs for security, compliance, and operational reasons such as to store the logs for an extended period of time, to stream the logs to other applications, repositories, or third parties, and to use the exported logs as input data for big-data analysis tools.


Audit

To determine if there are sinks created to export all the log entries generated within your Google Cloud projects, perform the following actions:

Using GCP Console

01 Sign in to the Google Cloud Management Console.

02 Select the GCP project that you want to examine from the console top navigation bar.

03 Navigate to Google Cloud Logging console at https://console.cloud.google.com/logs.

04 In the left navigation panel, select Log Router to access all the log sinks created for the selected GCP project.

05 Select the log sink that you want to examine, click on the 3-dot button to open the sink options menu, and select View sink details.

06 In the Sink details information box, check for the Inclusion filter configured for the selected sink. The Inclusion filter defines which log entries to include in logs routing sink. If the Inclusion filter box is not empty, the selected log sink is not configured to export all the log entries created for the selected GCP project.

07 Repeat steps no. 5 and 6 for all the log sinks listed in the Log Router Sinks section. If none of these sinks are using blank (empty) inclusion filters, there are no log sinks configured to export all the log entries for the selected Google Cloud Platform (GCP) project.

08 Repeat steps no. 2 – 7 for each GCP project deployed within your Google Cloud account.

Using GCP CLI

01 Run projects list command (Windows/macOS/Linux) using custom query filters to list the IDs of all the projects available in your Google Cloud account:

gcloud projects list
  --format="table(projectId)"

02 The command output should return the requested GCP project IDs:

PROJECT_ID
cc-project5-123123
cc-internal-111222
cc-web-prod-123456

03 Run logging sinks list command (Windows/macOS/Linux) using the ID of the GCP project that you want to examine as the identifier parameter and custom query filters to describe the name and the filter for each log sink created within the selected GCP project:

gcloud logging sinks list
  --project=cc-project5-123123
  --format="json(name,filter)"

04 The command output should return the requested log sink information:

[
	{
		"name": "cc-data-access-log-sink",
		"filter": "resource.type=\"project\"\nresource.labels.project_id=\"cc-project5-123123\"\nlogName=\"projects/cc-project5-123123/logs/cloudaudit.googleapis.com%2Fdata_access\""
	},
	{
		"name": "cc-service-account-sink",
		"filter": "resource.type=\"service_account\"\nresource.labels.email_id=\"retreterter@cc-project5-123123.iam.gserviceaccount.com\""
	}
]

Check the "filter" attribute value for each log sink object returned by the logging sinks list command output. If none of the sinks returned are using blank filters, i.e. the "filter" attribute value is not set to "(empty filter)", there are no log sinks configured to export all the log entries created for the selected Google Cloud Platform (GCP) project.

05 Repeat steps no. 3 and 4 for each GCP project deployed within your Google Cloud account.

Remediation / Resolution

To create and configure log routing sinks that export all the log entries generated within your Google Cloud projects, perform the following actions:

Using GCP Console

01 Sign in to the Google Cloud Management Console.

02 Select the GCP project that you want to examine from the console top navigation bar.

03 Navigate to Google Cloud Logging console at https://console.cloud.google.com/logs.

04 In the left navigation panel, select Log Router, and choose Create sink from the top menu to create a new log router sink for the selected GCP project.

05 On the Create logs routing sink setup page, perform the following operations:

  1. For Sink details, provide a name and description for the new logs routing sink. Choose NEXT to continue the setup.
  2. For Sink destination, click inside the Select sink service box, and select the service type for the new sink. The service selected at this step will store the exported logs. As an example, this conformity rule will use the Cloud Storage service to send the logs to a logging bucket. However, sinks can be configured to export logs to BigQuery datasets, Pub/Sub topics, or custom destinations. For logs destination, select the destination bucket. If you need to create a new bucket for your logs, choose the Create new log bucket option from the Select a log bucket dropdown menu, then follow the setup wizard to create a new destination bucket. Choose NEXT to continue.
  3. For Choose logs to include in sink, leave the Build inclusion filter box empty to skip defining an inclusion filter for the logs sink. If an inclusion filter is not provided for the sink, all ingested logs will be routed to the destination configured above. Choose NEXT to continue the setup.
  4. For Choose logs to filter out of sink (optional), skip configuring exclusion filters in order to export all the log entries generated within the selected GCP project.
  5. Choose Create sink to create your new logs routing sink. If the request is successful, the following confirmation message should be displayed: Your log sink was successfully created. Data should be available soon..

06 Repeat steps no. 2 – 5 to create logs routing sinks for other GCP projects deployed in your Google Cloud account.

Using GCP CLI

01 Create the storage resource that will be used as destination for the exported logs. As example, this conformity rule will use a Google Cloud Storage bucket as logs destination. However, log sinks can be configured to export logs to BigQuery datasets, Pub/Sub topics, or custom destinations. Run gsutil mb command (using gsutil tool) to create the Cloud Storage bucket where the log sink will export all the log entries created for the selected GCP project. The following example, creates a standard bucket named "cc-project5-logs-bucket", within the US East-1 region, for a project named "cc-project5-123123":

gsutil mb -p cc-project5-123123 -c STANDARD -l US-EAST1 -b on gs://cc-project5-logs-bucket/

02 The command output should return the gsutil mb request status:

Creating gs://cc-project5-logs-bucket/...

03 Run logging sinks create command (Windows/macOS/Linux) to create the logs routing sink required to export all the log entries generated for the selected Google Cloud project, to the destination bucket created at the previous steps:

gcloud logging sinks create cc-project5-log-sink storage.googleapis.com/cc-project5-logs-bucket

04 Type Y to confirm that the new sink is configured with an empty filter that matches all logs:

Sink with empty filter matches all entries.
Do you want to continue (Y/n)?  Y

05 The command request should return the information available for the new sink (including the sink URL and the writer identity ID):

Created [https://logging.googleapis.com/v2/projects/cc-project5-123123/sinks/cc-project5-log-sink].
Please remember to grant `serviceAccount:abcdabcdabcd-123123@gcp-sa-logging.iam.gserviceaccount.com` the Storage Object Creator role on the bucket.
More information about sinks can be found at https://cloud.google.com/logging/docs/export/configure_export

06 Run gsutil iam ch command (using gsutil Python tool) to add the sink's writer identity returned at the previous step to the destination bucket, and give this identity the Storage Object Creator role in order to allow it to create objects within that bucket (the command does not produce an output):

gcloud iam ch serviceAccount:abcdabcdabcd-123123@gcp-sa-logging.iam.gserviceaccount.com:objectCreator gs://cc-project5-logs-bucket

07 Repeat steps no. 1 – 6 to create logs routing sinks for other GCP projects created within your Google Cloud account.

References

Publication date Oct 28, 2021

Unlock the Remediation Steps


Free 30-day Trial

Automatically audit your configurations with Conformity
and gain access to our cloud security platform.

Confirmity Cloud Platform

No thanks, back to article

You are auditing:

Export All Log Entries Using Sinks

Risk Level: Medium