DuckDB
DuckDB reads every Send2Bucket destination in place: bucket objects, Iceberg tables and DuckLake tables. Nothing is copied first, so a query sees deliveries as soon as they land. Use it from the CLI, Python, Node.js, a notebook, or as the engine behind another tool.
We ran this page against S3-compatible storage, an Iceberg REST catalog and DuckLake. The Amazon S3, Google Cloud Storage, S3 Tables and Supabase sections follow DuckDB's documentation.
Records follow Delivery format. Use DuckDB 2.0 or later. Earlier versions read bucket objects and Iceberg tables, but not DuckLake catalogs that Send2Bucket writes.
Connect
Each section below ends with a view named records, with the same columns whatever the destination: signal, received_at, resource and data. The example queries run against that view.
Amazon S3
INSTALL httpfs; LOAD httpfs;
-- Uses your AWS profile, environment variables or instance role.
CREATE SECRET s2b_bucket (TYPE s3, PROVIDER credential_chain, REGION 'us-east-1');
CREATE VIEW records AS
SELECT * FROM read_json(
's3://<bucket>/<prefix>/s2b/stream=*/date=*/hour=*/*.ndjson.gz',
format = 'newline_delimited',
hive_partitioning = true,
columns = {v: 'INTEGER', delivery_batch_id: 'VARCHAR', batch_id: 'VARCHAR',
received_at: 'TIMESTAMPTZ', signal: 'VARCHAR',
resource: 'JSON', scope: 'JSON', data: 'JSON'});
hive_partitioning adds stream, date and hour columns from the path. Filter on them to skip objects: WHERE date = '2026-11-19' reads one day's folders only.
Listing columns keeps every type stable. Without it, DuckDB infers them from a sample of lines, and your JSON events may infer differently from one day to the next.
To read a single stream, replace stream=* with stream=<stream_id>.
Google Cloud Storage
Create an HMAC key for a service account that can read the bucket:
INSTALL httpfs; LOAD httpfs;
CREATE SECRET s2b_bucket (TYPE gcs, KEY_ID '<hmac_access_id>', SECRET '<hmac_secret>');
Then create the view as for Amazon S3, with gs://<bucket>/<prefix>/s2b/stream=*/date=*/hour=*/*.ndjson.gz.
S3-compatible storage
Add the endpoint to the secret. For Cloudflare R2:
CREATE SECRET s2b_bucket (
TYPE s3, KEY_ID '<access_key_id>', SECRET '<secret_access_key>',
REGION 'auto', ENDPOINT '<account_id>.r2.cloudflarestorage.com');
For MinIO and other self-hosted stores, also set URL_STYLE 'path', and USE_SSL false if the endpoint is plain HTTP. Then create the view as for Amazon S3.
Apache Iceberg (REST catalog)
INSTALL httpfs; LOAD httpfs; INSTALL iceberg; LOAD iceberg;
-- Storage for the table's data files.
CREATE SECRET s2b_storage (TYPE s3, KEY_ID '<access_key_id>', SECRET '<secret_access_key>', REGION '<region>');
CREATE SECRET s2b_catalog (TYPE iceberg, TOKEN '<catalog_token>');
ATTACH '<warehouse>' AS lakehouse (TYPE iceberg, ENDPOINT '<catalog_uri>', SECRET s2b_catalog);
CREATE VIEW records AS
SELECT signal_type AS signal, ingested_at AS received_at, delivery_batch_id,
payload::JSON->'$.resource' AS resource, payload::JSON->'$.data' AS data
FROM lakehouse.<namespace>.<table>;
Use a read-only token and storage key here, not the ones Send2Bucket writes with.
Amazon S3 Tables
INSTALL httpfs; LOAD httpfs; INSTALL iceberg; LOAD iceberg;
CREATE SECRET s2b_storage (TYPE s3, PROVIDER credential_chain, REGION 'us-east-1');
ATTACH 'arn:aws:s3tables:us-east-1:123456789012:bucket/analytics' AS lakehouse
(TYPE iceberg, ENDPOINT_TYPE s3_tables);
Then create the view as for Iceberg, from lakehouse.<namespace>.<table>. Your credentials need s3tables:Get* and s3tables:List* on the table bucket, and s3:GetObject on the table's data.
Supabase Analytics Buckets
INSTALL httpfs; LOAD httpfs; INSTALL iceberg; LOAD iceberg;
CREATE SECRET s2b_storage (
TYPE s3, KEY_ID '<access_key_id>', SECRET '<secret_access_key>', REGION '<region>',
ENDPOINT '<project_ref>.supabase.co/storage/v1/s3', URL_STYLE 'path');
CREATE SECRET s2b_catalog (TYPE iceberg, TOKEN '<catalog_token>');
ATTACH '<analytics_bucket>' AS lakehouse (
TYPE iceberg, ENDPOINT 'https://<project_ref>.supabase.co/storage/v1/iceberg', SECRET s2b_catalog);
Then create the view as for Iceberg.
DuckLake
INSTALL httpfs; LOAD httpfs; INSTALL ducklake; LOAD ducklake;
CREATE SECRET s2b_storage (TYPE s3, KEY_ID '<access_key_id>', SECRET '<secret_access_key>', REGION '<region>');
ATTACH 'ducklake:postgres:host=db.example.com dbname=lake user=analyst password=<password>' AS lake
(METADATA_SCHEMA '<metadata_schema>', READ_ONLY);
CREATE VIEW records AS
SELECT signal_type AS signal, ingested_at AS received_at, delivery_batch_id,
payload::JSON->'$.resource' AS resource, payload::JSON->'$.data' AS data
FROM lake.<namespace>.<table>;
Omit METADATA_SCHEMA if you did not set one on the destination. READ_ONLY keeps an analyst session from committing to the lake. Each delivery batch is one DuckLake snapshot; SELECT * FROM ducklake_snapshots('lake') lists them.
Example queries
Records per signal:
SELECT signal, count(*) AS records
FROM records
GROUP BY ALL
ORDER BY records DESC;
JSON events by name, with revenue from order.paid:
SELECT data->>'$.event' AS event,
count(*) AS events,
sum((data->>'$.cents')::INTEGER) / 100 AS revenue_usd
FROM records
WHERE signal = 'event'
GROUP BY ALL
ORDER BY events DESC;
┌────────────┬────────┬─────────────┐
│ event │ events │ revenue_usd │
├────────────┼────────┼─────────────┤
│ page.view │ 177 │ NULL │
│ order.paid │ 53 │ 4161.0 │
│ email.sent │ 47 │ NULL │
│ signup │ 23 │ NULL │
└────────────┴────────┴─────────────┘
Log records by service and severity. service.name is one entry in the resource's attribute list:
SELECT (SELECT a->>'$.value.stringValue'
FROM unnest(json_extract(resource, '$.attributes[*]')) AS t(a)
WHERE a->>'$.key' = 'service.name') AS service,
data->>'$.severityText' AS severity,
count(*) AS logs
FROM records
WHERE signal = 'log'
GROUP BY ALL;
The five slowest spans. Timestamps are nanosecond strings, so cast to HUGEINT before subtracting:
SELECT data->>'$.name' AS span,
data->>'$.traceId' AS trace_id,
((data->>'$.endTimeUnixNano')::HUGEINT - (data->>'$.startTimeUnixNano')::HUGEINT) / 1e6 AS duration_ms
FROM records
WHERE signal = 'span'
ORDER BY duration_ms DESC
LIMIT 5;
Metrics by name and type, and gauge data points:
SELECT data->>'$.name' AS metric,
list_filter(json_keys(data),
lambda k: k IN ('gauge', 'sum', 'histogram', 'exponentialHistogram', 'summary'))[1] AS type,
count(*) AS metrics
FROM records
WHERE signal = 'metric'
GROUP BY ALL;
SELECT data->>'$.name' AS metric,
make_timestamp_ns((p->>'$.timeUnixNano')::BIGINT) AS at,
coalesce((p->>'$.asDouble')::DOUBLE, (p->>'$.asInt')::BIGINT) AS value
FROM records, unnest(json_extract(data, '$.gauge.dataPoints[*]')) AS u(p)
WHERE signal = 'metric';
Export to Parquet
Bucket destinations hold gzip JSON, which every engine reads but none reads fast. For heavy repeated queries, copy a day to Parquet and query that:
COPY (SELECT * EXCLUDE (resource, scope) FROM records WHERE signal = 'event' AND date = '2026-11-19')
TO 's3://<bucket>/parquet/events/date=2026-11-19.parquet' (FORMAT parquet);
If you query this way every day, a table destination such as DuckLake or Apache Iceberg stores Parquet from the start.
Notes
- DuckDB holds no copy of your data between sessions unless you write one. Put the
CREATE SECRET,ATTACHandCREATE VIEWstatements in a file and run it at the start of each session, or useCREATE PERSISTENT SECRETin a DuckDB database file. - MotherDuck runs the same SQL in the cloud and shares it with your team.
- To load records into another database on a schedule, DuckDB is the loader in the PostgreSQL and MySQL guides.