Skip to Content

Coalesce Stackdriver logs into one BigQuery dataset

How to coalesce your Stackdriver logs / sinks into a single BigQuery project

Posted on One min read

Exporting Stackdriver logs back to BigQuery for analysis is a very straightforward task (and a very good pattern to follow too, by the way).

However, when you are managing lots of projects, you’ll probably want to coalesce the logs from all your projects, into one single dedicated BigQuery project/dataset, for easier querying and maintainability.

However, setting this up in the cloud console isn’t that obvious to most users. This is because the configuration in the Stackdriver sink config doesn’t actually allow you to select a different project/dataset to send the logs to. It only lets you select the currently selected/working project:

The trick is to select Custom destination from the drop-down, and then use the following format to specify the project / dataset where you want the logs to go. Plug in the project and dataset that you want to use:

bigquery.googleapis.com/projects/[PROJECT_ID]/datasets/[DATASET_ID]

After you’ve done that, all you need to do is to give the sink writer editor permissions on the said project / dataset, and you’re done.

Easy-peasy-lemon-squeezy!

For more information

To learn more, you can have a look at this question on StackOverflow which covered the topic, and the Stackdriver logging export documentation.

comments powered by Disqus