Skip to main content

DuckLake

Send2Bucket appends rows to a DuckLake table whose metadata catalog is a PostgreSQL database you run. Each delivery batch is one DuckLake transaction.

What you provide​

FieldKindExample
data_pathsettings3://analytics-lake/ducklake/. Used when the lake is created; an existing lake keeps its own.
metadata_schemasetting, optionalThe Postgres schema holding DuckLake's metadata tables
namespacesettingThe DuckLake schema for the table. Defaults to main.
tablesettingevents
catalog_dsncredentialA libpq string: host=db.example.com dbname=lake user=s2bucket password=…
access_key_id, secret_access_keycredentialAn S3 key that can read and write under the data path
regioncredentialThe data path's region, for example us-east-1
endpointcredential, optionalFor S3-compatible storage outside AWS, for example https://<account_id>.r2.cloudflarestorage.com

The catalog must be reachable from Send2Bucket over TLS. Create a dedicated database user for Send2Bucket, with rights only on the catalog database.

Setup​

  1. Create the catalog database and the user:

    CREATE USER s2bucket WITH PASSWORD '<password>';
    CREATE DATABASE lake OWNER s2bucket;
  2. Add the destination in the dashboard and select Verify. Send2Bucket attaches the catalog, creates the table in the namespace if it is missing, or checks an existing table's columns.

The table has the columns in Delivery format. An existing table with other columns is left untouched, and setup fails with incompatible table: ….

Delivery and retries​

A batch is delivered when the DuckLake transaction that appends its rows commits. If the commit response is lost, Send2Bucket looks for the batch's delivery_batch_id in the table before trying again, so a retry does not append the rows twice.

Reading the table​

INSTALL ducklake;
ATTACH 'ducklake:postgres:dbname=lake host=db.example.internal user=analyst' AS lake
(DATA_PATH 's3://analytics-lake/ducklake/');
SELECT signal_type, count(*) FROM lake.main.events GROUP BY ALL;

Query your data​

  • DuckDB: attaches the lake read-only.
  • PostgreSQL: through pg_duckdb.
  • MySQL: load with DuckDB.
  • Python: Polars, PyIceberg or the duckdb package.
  • R: the duckdb package.

To explore in a notebook or BI tool, see Integrations. The querying overview compares engines.