Skip to main content

Google Cloud Storage

Send2Bucket writes to Cloud Storage with its own Google service account. You grant that account access to your bucket; no keys change hands.

What you provide​

  • Settings: bucket and prefix, for example analytics-raw and raw.
  • Send2Bucket's service account email, shown in the dashboard on the destination.

1. Grant access​

Send2Bucket needs to create objects and to read the objects it wrote. Grant both roles on the bucket:

SA=<gcs_service_account>

gcloud storage buckets add-iam-policy-binding gs://analytics-raw \
--member="serviceAccount:$SA" --role="roles/storage.objectCreator"

gcloud storage buckets add-iam-policy-binding gs://analytics-raw \
--member="serviceAccount:$SA" --role="roles/storage.objectViewer"

To limit access to the prefix, add an IAM condition to each binding. Conditions need uniform bucket-level access.

--condition='expression=resource.name.startsWith("projects/_/buckets/analytics-raw/objects/raw/s2b/"),title=s2bucket-prefix'

2. Add the destination​

Enter the bucket and the prefix in the dashboard and select Verify. Send2Bucket writes a test object under raw/s2b/ and reads it back; the destination then becomes active.

How uploads are made safe to retry​

Send2Bucket creates each object with an ifGenerationMatch=0 precondition, so an upload never replaces an existing object. After an ambiguous response it reads the existing object and checks that it holds the same delivery batch before marking the batch delivered.

Objects use the layout in Delivery format.

Query your data​

To explore in a notebook or BI tool, see Integrations. The querying overview compares engines.