Google Cloud Storage
Send2Bucket writes to Cloud Storage with its own Google service account. You grant that account access to your bucket; no keys change hands.
What you provide
- Settings:
bucketandprefix, for exampleanalytics-rawandraw. - Send2Bucket's service account email, shown in the dashboard on the destination.
1. Grant access
Send2Bucket needs to create objects and to read the objects it wrote. Grant both roles on the bucket:
SA=<gcs_service_account>
gcloud storage buckets add-iam-policy-binding gs://analytics-raw \
--member="serviceAccount:$SA" --role="roles/storage.objectCreator"
gcloud storage buckets add-iam-policy-binding gs://analytics-raw \
--member="serviceAccount:$SA" --role="roles/storage.objectViewer"
To limit access to the prefix, add an IAM condition to each binding. Conditions need uniform bucket-level access.
--condition='expression=resource.name.startsWith("projects/_/buckets/analytics-raw/objects/raw/s2b/"),title=s2bucket-prefix'
2. Add the destination
Enter the bucket and the prefix in the dashboard and select Verify. Send2Bucket writes a test object under raw/s2b/ and reads it back; the destination then becomes active.
How uploads are made safe to retry
Send2Bucket creates each object with an ifGenerationMatch=0 precondition, so an upload never replaces an existing object. After an ambiguous response it reads the existing object and checks that it holds the same delivery batch before marking the batch delivered.
Objects use the layout in Delivery format.
Query your data
- BigQuery: external table or scheduled load.
- DuckDB: reads the objects in place with an HMAC key.
- Trino: external table over the bucket.
- PostgreSQL: load into
jsonb. - MySQL: load into
JSON. - Python: Polars, PyIceberg or the duckdb package.
- R: the duckdb package.
- ClickHouse
- Snowflake
- Microsoft Fabric
- Databricks
- Apache Spark
To explore in a notebook or BI tool, see Integrations. The querying overview compares engines.