Skip to main content

Querying your data

Send2Bucket stops at your bucket or table. Everything after that runs on your side, with the engine you already use: there is no query API in Send2Bucket and nothing to wire up in the dashboard. These pages show how to connect common SQL engines and run the same questions in each.

What you are querying​

DestinationYou getRead it with
Amazon S3, Google Cloud Storage, S3-compatiblegzip NDJSON objects under <prefix>/s2b/stream=…/date=…/hour=…/Any engine that reads JSON files from object storage
Apache Iceberg, Amazon S3 Tables, Supabase Analytics BucketsAn Iceberg tableAny Iceberg client pointed at the same catalog
DuckLakeA DuckLake tableDuckDB, or Postgres through pg_duckdb

Every record carries the same envelope (Delivery format):

  • Objects have one envelope per line, with top-level signal, received_at, resource, scope and data fields. stream, date and hour come from the path.
  • Tables have signal_type, delivery_batch_id, ingested_at and payload, where payload is the envelope as JSON text. A JSON event's fields are at $.data.<field>.

Engines​

EngineBucket objectsIcebergDuckLakePage
DuckDBReads in placeReads in placeReads in placeDuckDB
Trino, StarburstExternal tableIceberg connectorNoTrino
BigQueryExternal tableBigLake Iceberg tableNoBigQuery
PostgreSQLLoad into jsonb, or Supabase WrappersLoad, or Supabase WrappersLoadPostgreSQL
MySQLLoad into JSONLoadLoadMySQL
ClickHouseReads in place, or loads into MergeTreeDataLakeCatalog or icebergS3NoClickHouse
Amazon AthenaExternal table with partition projectionGlue or S3 Tables catalogNoAmazon Athena
Amazon RedshiftSpectrum external table, or COPY into SUPERS3 Tables through GlueNoAmazon Redshift
Apache SparkReads in place (JSON reader)Iceberg REST catalogNoApache Spark
Databricksread_files, or Auto Loader into DeltaNo generic REST catalogNoDatabricks
StarRocksFILES() over lines with parse_json, or a Pipe into a native tableExternal Iceberg catalogNoStarRocks
DremioS3 source, folder promoted as JSONIceberg REST source (Enterprise)NoDremio
SnowflakeExternal table, or Snowpipe or COPY into VARIANTCatalog integration (REST, S3 Tables)NoSnowflake
Microsoft FabricOneLake shortcut, Spark load into DeltaIceberg shortcut, with limitsNoMicrosoft Fabric
Python (Polars, PyIceberg, duckdb)Reads in placeReads in placeReads in placePython
R (duckdb, arrow)Reads in placeReads in placeReads in placeR

"Reads in place" means the engine queries your bucket or catalog directly and always sees the latest deliveries. "Load" means you copy delivered data into the database on a schedule. Each object and each table commit has its own delivery_batch_id, so a load job can skip the ones it already copied.

The example questions​

Every engine page answers the same four questions so you can compare them:

  1. How many records of each signal arrived?
  2. Which JSON events are most frequent, and how much revenue did order.paid events carry?
  3. How many log records did each service send, by severity?
  4. Which spans were slowest?

The examples use a stream that receives product events such as:

{"event":"order.paid","user_id":"usr_0012","plan":"pro","order_id":"ord_20931","cents":4900,"currency":"usd","items":[{"sku":"sku_1","qty":2}]}

and OpenTelemetry logs, traces and metrics from a service named checkout. Swap the field names for your own.

Working with OpenTelemetry records​

OTLP records use the OTLP JSON mapping:

  • Attributes are a list of {"key": …, "value": {"stringValue": …}} objects, not a map. To read service.name, search resource.attributes for the matching key. Each engine page shows how.
  • 64-bit integers, such as startTimeUnixNano, are strings. Cast them to a 64-bit or wider integer before doing arithmetic.
  • A metric's data holds one of gauge, sum, histogram, exponentialHistogram or summary, each with a dataPoints list.

Tips​

  • Filter on date (objects) or ingested_at (tables) first. Engines skip whole partitions or files for those filters.
  • Read only stream=* folders under <prefix>/s2b/. Destination checks write small test objects to <prefix>/s2b/_verify/, which are not records.
  • A record is in your bucket or table about a minute after it was accepted. See Delivery semantics for when duplicates can happen and how to drop them.

BI tools and notebooks​

To explore data in a hosted notebook or BI tool, connect it to one of the engines above. See Integrations.