Querying your data
Send2Bucket stops at your bucket or table. Everything after that runs on your side, with the engine you already use: there is no query API in Send2Bucket and nothing to wire up in the dashboard. These pages show how to connect common SQL engines and run the same questions in each.
What you are querying
| Destination | You get | Read it with |
|---|---|---|
| Amazon S3, Google Cloud Storage, S3-compatible | gzip NDJSON objects under <prefix>/s2b/stream=…/date=…/hour=…/ | Any engine that reads JSON files from object storage |
| Apache Iceberg, Amazon S3 Tables, Supabase Analytics Buckets | An Iceberg table | Any Iceberg client pointed at the same catalog |
| DuckLake | A DuckLake table | DuckDB, or Postgres through pg_duckdb |
Every record carries the same envelope (Delivery format):
- Objects have one envelope per line, with top-level
signal,received_at,resource,scopeanddatafields.stream,dateandhourcome from the path. - Tables have
signal_type,delivery_batch_id,ingested_atandpayload, wherepayloadis the envelope as JSON text. A JSON event's fields are at$.data.<field>.
Engines
| Engine | Bucket objects | Iceberg | DuckLake | Page |
|---|---|---|---|---|
| DuckDB | Reads in place | Reads in place | Reads in place | DuckDB |
| Trino, Starburst | External table | Iceberg connector | No | Trino |
| BigQuery | External table | BigLake Iceberg table | No | BigQuery |
| PostgreSQL | Load into jsonb, or Supabase Wrappers | Load, or Supabase Wrappers | Load | PostgreSQL |
| MySQL | Load into JSON | Load | Load | MySQL |
| ClickHouse | Reads in place, or loads into MergeTree | DataLakeCatalog or icebergS3 | No | ClickHouse |
| Amazon Athena | External table with partition projection | Glue or S3 Tables catalog | No | Amazon Athena |
| Amazon Redshift | Spectrum external table, or COPY into SUPER | S3 Tables through Glue | No | Amazon Redshift |
| Apache Spark | Reads in place (JSON reader) | Iceberg REST catalog | No | Apache Spark |
| Databricks | read_files, or Auto Loader into Delta | No generic REST catalog | No | Databricks |
| StarRocks | FILES() over lines with parse_json, or a Pipe into a native table | External Iceberg catalog | No | StarRocks |
| Dremio | S3 source, folder promoted as JSON | Iceberg REST source (Enterprise) | No | Dremio |
| Snowflake | External table, or Snowpipe or COPY into VARIANT | Catalog integration (REST, S3 Tables) | No | Snowflake |
| Microsoft Fabric | OneLake shortcut, Spark load into Delta | Iceberg shortcut, with limits | No | Microsoft Fabric |
| Python (Polars, PyIceberg, duckdb) | Reads in place | Reads in place | Reads in place | Python |
| R (duckdb, arrow) | Reads in place | Reads in place | Reads in place | R |
"Reads in place" means the engine queries your bucket or catalog directly and always sees the latest deliveries. "Load" means you copy delivered data into the database on a schedule. Each object and each table commit has its own delivery_batch_id, so a load job can skip the ones it already copied.
The example questions
Every engine page answers the same four questions so you can compare them:
- How many records of each signal arrived?
- Which JSON events are most frequent, and how much revenue did
order.paidevents carry? - How many log records did each service send, by severity?
- Which spans were slowest?
The examples use a stream that receives product events such as:
{"event":"order.paid","user_id":"usr_0012","plan":"pro","order_id":"ord_20931","cents":4900,"currency":"usd","items":[{"sku":"sku_1","qty":2}]}
and OpenTelemetry logs, traces and metrics from a service named checkout. Swap the field names for your own.
Working with OpenTelemetry records
OTLP records use the OTLP JSON mapping:
- Attributes are a list of
{"key": …, "value": {"stringValue": …}}objects, not a map. To readservice.name, searchresource.attributesfor the matching key. Each engine page shows how. - 64-bit integers, such as
startTimeUnixNano, are strings. Cast them to a 64-bit or wider integer before doing arithmetic. - A metric's
dataholds one ofgauge,sum,histogram,exponentialHistogramorsummary, each with adataPointslist.
Tips
- Filter on
date(objects) oringested_at(tables) first. Engines skip whole partitions or files for those filters. - Read only
stream=*folders under<prefix>/s2b/. Destination checks write small test objects to<prefix>/s2b/_verify/, which are not records. - A record is in your bucket or table about a minute after it was accepted. See Delivery semantics for when duplicates can happen and how to drop them.
BI tools and notebooks
To explore data in a hosted notebook or BI tool, connect it to one of the engines above. See Integrations.