Apache Iceberg (REST catalog)
Send2Bucket appends to an Iceberg table through any catalog that implements the Iceberg REST catalog protocol, managed or self-hosted. Each delivery batch is one append snapshot.
What you provide
| Field | Kind | Example |
|---|---|---|
catalog_uri | setting | https://catalog.example.com/api/catalog |
warehouse | setting | analytics |
namespace | setting | s2b (must exist) |
table | setting | events |
token | credential | A bearer token for the catalog |
access_key_id, secret_access_key | credential | An S3 key that can write the table's data files |
region | credential | The data location's region |
endpoint | credential, optional | For S3-compatible storage outside AWS |
Scope the token to the namespace: Send2Bucket needs to create and load the table and to commit appends. Scope the storage key to the table's data location.
Setup
- Create the namespace in your catalog.
- Add the destination in the dashboard and select Verify. Send2Bucket creates the table with the columns in Delivery format if it is missing. Repeating setup, or two setups racing, reuses a compatible table.
- An existing table with other columns is left untouched, and setup fails with
incompatible table: ….
Send2Bucket follows the table's schema and partition spec. You can add a partition spec, for example on day(ingested_at), before the first delivery.
Delivery and retries
A batch is delivered when the catalog confirms the append snapshot. If that response is lost, Send2Bucket checks the table for the batch's delivery_batch_id before retrying, so a batch is committed once. Appends from your own jobs to the same table are kept.
Query your data
- DuckDB: attaches the catalog.
- Trino: Iceberg connector with a REST catalog.
- BigQuery: BigLake Iceberg table, which lags behind new commits.
- PostgreSQL: pg_duckdb, or load.
- MySQL: load with DuckDB.
- Python: Polars, PyIceberg or the duckdb package.
- R: the duckdb package.
- ClickHouse
- Snowflake
- Microsoft Fabric
- Apache Spark
- StarRocks
- Dremio
To explore in a notebook or BI tool, see Integrations. The querying overview compares engines.