Skip to Content

Faster Uploads to Cloud Storage

Using gsutil's -m parallel flag

Posted on One min read

For our first tip towards 2018, we’ll have a quick look at Google Cloud Storage. With GCS, for short, you can use the gsutil command-line utility to upload all your files and data in the cloud easily.

There are different levels of storages classes, like multi-regional, regional, nearline or coldline. Regardless of the level you chose, whenever you need to upload a big number of files from your machine, it can take a little while to upload files.

When uploading a single file, you can do:

$ gsutil cp file1.txt gs://my-text-notes

With multiple files, you can use some wildcards, as you can expect:

$ gsutil cp *.txt gs://my-text-notes

Now, let’s imagine that you have thousands of text files, it might actually take some time to upload, as files are pushed individually, serially, one after the other. But fortunately, here’s our first trick: the -m flag, to send files in parallel! Let’s now use our magic flag, and upload files in a parallel fashion with:

$ gsutil -m cp *.txt gs://my-text-notes
comments powered by Disqus