Skip to main content

Apache Iceberg (REST catalog)

Send2Bucket appends to an Iceberg table through any catalog that implements the Iceberg REST catalog protocol, managed or self-hosted. Each delivery batch is one append snapshot.

What you provide​

FieldKindExample
catalog_urisettinghttps://catalog.example.com/api/catalog
warehousesettinganalytics
namespacesettings2b (must exist)
tablesettingevents
tokencredentialA bearer token for the catalog
access_key_id, secret_access_keycredentialAn S3 key that can write the table's data files
regioncredentialThe data location's region
endpointcredential, optionalFor S3-compatible storage outside AWS

Scope the token to the namespace: Send2Bucket needs to create and load the table and to commit appends. Scope the storage key to the table's data location.

Setup​

  1. Create the namespace in your catalog.
  2. Add the destination in the dashboard and select Verify. Send2Bucket creates the table with the columns in Delivery format if it is missing. Repeating setup, or two setups racing, reuses a compatible table.
  3. An existing table with other columns is left untouched, and setup fails with incompatible table: ….

Send2Bucket follows the table's schema and partition spec. You can add a partition spec, for example on day(ingested_at), before the first delivery.

Delivery and retries​

A batch is delivered when the catalog confirms the append snapshot. If that response is lost, Send2Bucket checks the table for the batch's delivery_batch_id before retrying, so a batch is committed once. Appends from your own jobs to the same table are kept.

Query your data​

To explore in a notebook or BI tool, see Integrations. The querying overview compares engines.