Skip to main content

Delivery format

All bucket destinations (Amazon S3, Google Cloud Storage and S3-compatible storage) get the same object layout. Table destinations get the same four columns.

Object keys​

Each delivery batch is one gzip-compressed NDJSON object:

<prefix>/s2b/stream=<stream_id>/date=YYYY-MM-DD/hour=HH/<delivery_batch_id>.ndjson.gz
  • <prefix> is the prefix you configured for the destination.
  • <stream_id> is the stream's UUID.
  • date and hour are the UTC hour in which Send2Bucket received the records.
  • A delivery batch is written when it reaches 16 MiB of uncompressed data or 60 seconds after it opened, whichever comes first. A quiet stream is still delivered on the timer.
  • The key and the contents of an object never change between retries. A retry writes identical bytes under the same key, or confirms that the object already written belongs to the same delivery batch.

The key=value path segments work as Hive-style partitions in DuckDB, Spark, Athena and other engines.

The envelope​

Each line is one record wrapped in an envelope:

{"v":1,"delivery_batch_id":"dlv_01jd3kb7m4","batch_id":"bat_01jd3k9qx2","received_at":"2026-11-19T03:04:05.123456Z","signal":"event","data":{"event":"order.paid","order_id":"ord_20931","cents":4200}}
FieldTypeMeaning
vnumberEnvelope version, currently 1.
delivery_batch_idstringThe object or table commit this line belongs to (dlv_…).
batch_idstringThe batch that carried the record (bat_…): one per request, or one per stream in the batch format.
received_atstringWhen Send2Bucket accepted the request, RFC 3339 in UTC with microseconds.
signalstringevent, log, span or metric.
resourceobjectOTLP only. The resource of the record.
scopeobjectOTLP only. The instrumentation scope of the record.
dataobjectThe record.

JSON events​

data is the object you sent, unchanged: same keys, same values, same nesting.

OpenTelemetry records​

Each log record, span or metric becomes one line. data is the OTLP JSON mapping of the LogRecord, Span or Metric, and resource and scope are copied onto every line, so a line can be read on its own. In that mapping:

  • 64-bit integers, such as startTimeUnixNano, are JSON strings, so no precision is lost.
  • Trace and span IDs are lowercase hex strings.
  • Field names are lowerCamelCase and enums are integers.
{"v":1,"delivery_batch_id":"dlv_01jd3kc2r8","batch_id":"bat_01jd3kbz51","received_at":"2026-11-19T03:04:07.481920Z","signal":"span","resource":{"attributes":[{"key":"service.name","value":{"stringValue":"checkout"}}]},"scope":{"name":"checkout.http","version":"1.4.0"},"data":{"traceId":"5b8efff798038103d269b633813fc60c","spanId":"eee19b7ec3c1b174","name":"POST /orders","kind":2,"startTimeUnixNano":"1763521447100000000","endTimeUnixNano":"1763521447181000000","attributes":[{"key":"http.response.status_code","value":{"intValue":"201"}}],"status":{}}}

Metrics keep their name, unit, type, temporality, monotonicity, exemplars and every data point, including histogram buckets, exponential histogram scales and summary quantiles.

Reading objects​

DuckDB reads the gzip files directly:

SELECT received_at, data->>'event' AS event, data->>'order_id' AS order_id
FROM read_json('s3://analytics-raw/raw/s2b/stream=*/date=*/hour=*/*.ndjson.gz',
hive_partitioning = true)
WHERE signal = 'event' AND date = '2026-11-19';

See Querying for DuckDB, Trino, BigQuery, PostgreSQL and MySQL.

Table columns​

DuckLake, Apache Iceberg, Amazon S3 Tables and Supabase Analytics Buckets get one row per record:

ColumnTypeValue
signal_typeVARCHARevent, log, span or metric
delivery_batch_idVARCHARThe delivery batch that committed the row
ingested_atTIMESTAMPTZWhen Send2Bucket accepted the record
payloadVARCHARThe envelope above, as JSON text

Query the payload with your engine's JSON functions:

SELECT ingested_at, json_extract_string(payload, '$.data.event') AS event
FROM events
WHERE signal_type = 'event';

Setup creates the table in the namespace you name, which must already exist. An existing table with these columns is reused. An existing table with other columns is left untouched, and setup fails with incompatible table: ….