Skip to main content

Python

note

We ran this page with Polars 2.1.0, PyIceberg 0.12.0 and DuckDB 2.0.0 against S3-compatible objects, an Iceberg REST catalog and DuckLake. The Amazon S3 Tables, Supabase Analytics, Google Cloud Storage and Cloudflare R2 settings follow the vendors' documentation.

Python reads every Send2Bucket destination without a database server. Use Polars for bucket objects, PyIceberg for Iceberg tables, and the duckdb package for DuckLake tables or when you would rather write SQL and get a pandas DataFrame. Each path below ends with a Polars LazyFrame named records, and the example queries run against it as Polars expressions.

pip install polars 'pyiceberg[pyarrow]' duckdb pandas

We ran the snippets on this page with Polars 2.1, PyIceberg 0.12 and DuckDB 2.0 against local S3-compatible storage, an Iceberg REST catalog and a DuckLake catalog in PostgreSQL. The Amazon S3 Tables, Supabase and Google Cloud Storage connection settings follow the vendors' documentation and were not run by us.

The envelope schema​

Polars infers a JSON schema from the first lines it reads. An object that holds only JSON events infers a data column without any span or log fields, and the OTLP lines that follow lose them. Declare the fields you query instead:

import polars as pl

ATTRIBUTES = pl.List(pl.Struct({"key": pl.String, "value": pl.Struct({"stringValue": pl.String})}))
RESOURCE = pl.Struct({"attributes": ATTRIBUTES})
DATA = pl.Struct({
# Your JSON event fields
"event": pl.String,
"order_id": pl.String,
"cents": pl.Int64,
# OTLP log and span fields
"severityText": pl.String,
"name": pl.String,
"traceId": pl.String,
"startTimeUnixNano": pl.String,
"endTimeUnixNano": pl.String,
})
ENVELOPE = {
"delivery_batch_id": pl.String,
"received_at": pl.String,
"signal": pl.String,
"resource": RESOURCE,
"data": DATA,
}

Fields you leave out are skipped when the lines are parsed, so DATA lists only what you read. OTLP 64-bit integers such as startTimeUnixNano arrive as JSON strings; keep them as pl.String here and cast when you do arithmetic.

Connect​

Amazon S3​

pl.scan_ndjson reads gzip NDJSON straight from the bucket:

SOURCE = "s3://<bucket>/<prefix>/s2b/stream=*/date=*/hour=*/*.ndjson.gz"

records = (
pl.scan_ndjson(SOURCE, schema=ENVELOPE, include_file_paths="path",
storage_options={"aws_region": "<region>"})
.with_columns(
pl.col("received_at").str.to_datetime(time_zone="UTC"),
stream=pl.col("path").str.extract(r"/stream=([^/]+)/"),
date=pl.col("path").str.extract(r"/date=([^/]+)/").str.to_date(),
hour=pl.col("path").str.extract(r"/hour=(\d+)/").cast(pl.Int8),
)
)
  • With no keys in storage_options, Polars reads credentials from the standard AWS_* environment variables and the instance or task role. For a named profile or an assumed role, pass credential_provider=pl.CredentialProviderAWS(profile_name="<profile>"), which needs boto3.
  • scan_ndjson has no hive_partitioning option, so the stream, date and hour columns above come from the object path. Filtering on them does not skip objects. To read less, narrow the glob instead: stream=<stream_id>/date=2026-11-19/hour=*/*.ndjson.gz reads one stream's day.
  • stream=* keeps the destination's test objects under s2b/_verify/ out of the scan.

Google Cloud Storage​

Use a gs://<bucket>/<prefix>/s2b/stream=*/date=*/hour=*/*.ndjson.gz source with the same code. Polars uses the service account in GOOGLE_APPLICATION_CREDENTIALS, or one you pass as storage_options={"service_account": "<path/to/key.json>"}.

S3-compatible storage​

Pass the endpoint and keys. For Cloudflare R2:

storage_options = {
"aws_endpoint_url": "https://<account_id>.r2.cloudflarestorage.com",
"aws_access_key_id": "<access_key_id>",
"aws_secret_access_key": "<secret_access_key>",
"aws_region": "auto",
}

For MinIO and other self-hosted stores over plain HTTP, also set "aws_allow_http": "true". Then build records as for Amazon S3 with these storage_options.

Apache Iceberg (REST catalog)​

PyIceberg loads the table from the catalog and reads its Parquet files:

from pyiceberg.catalog import load_catalog

catalog = load_catalog("s2b", **{
"type": "rest",
"uri": "<catalog_uri>",
"warehouse": "<warehouse>",
"token": "<catalog_token>",
"s3.access-key-id": "<access_key_id>",
"s3.secret-access-key": "<secret_access_key>",
"s3.region": "<region>",
})
table = catalog.load_table("<namespace>.<table>")

For S3-compatible storage, add "s3.endpoint": "https://<endpoint>". Use a read-only token and storage key, not the ones Send2Bucket writes with.

table.scan() takes a row filter, which PyIceberg uses to skip data files, and returns Arrow or Polars:

day = table.scan(row_filter="ingested_at >= '2026-11-19T00:00:00+00:00'")
arrow_table = day.to_arrow() # pyarrow.Table
frame = day.to_polars() # polars.DataFrame

The table's payload column is the envelope as JSON text. Decode it with the same schema to get the records columns:

def records_from_table(rows: pl.LazyFrame) -> pl.LazyFrame:
return (
rows.select(
signal="signal_type",
received_at="ingested_at",
delivery_batch_id="delivery_batch_id",
envelope=pl.col("payload").str.json_decode(pl.Struct({"resource": RESOURCE, "data": DATA})),
)
.unnest("envelope")
)

records = records_from_table(day.to_polars().lazy())

to_polars() reads the whole scan into memory. For a lazy scan that Polars plans itself, use pl.scan_iceberg(table, storage_options={...}) with the storage keys in Polars' aws_* form, as for bucket objects.

Amazon S3 Tables​

PyIceberg signs catalog requests with your AWS credentials (it needs boto3):

catalog = load_catalog("s3tables", **{
"type": "rest",
"uri": "https://s3tables.us-east-1.amazonaws.com/iceberg",
"warehouse": "arn:aws:s3tables:us-east-1:123456789012:bucket/analytics",
"rest.sigv4-enabled": "true",
"rest.signing-name": "s3tables",
"rest.signing-region": "us-east-1",
})

Then load the table and build records as for Iceberg. Your credentials need s3tables:Get* and s3tables:List* on the table bucket.

Supabase Analytics Buckets​

catalog = load_catalog("supabase", **{
"type": "rest",
"uri": "https://<project_ref>.supabase.co/storage/v1/iceberg",
"warehouse": "<analytics_bucket>",
"token": "<catalog_token>",
"s3.endpoint": "https://<project_ref>.supabase.co/storage/v1/s3",
"s3.access-key-id": "<access_key_id>",
"s3.secret-access-key": "<secret_access_key>",
"s3.region": "<region>",
"s3.force-virtual-addressing": False,
})

Then load the table and build records as for Iceberg.

DuckLake​

PyIceberg and Polars do not read DuckLake catalogs. Attach the lake with the duckdb package (2.0 or later) and hand the rows to Polars:

import duckdb

duckdb.sql("""
INSTALL ducklake; LOAD ducklake;
CREATE SECRET s2b_storage (TYPE s3, KEY_ID '<access_key_id>', SECRET '<secret_access_key>', REGION '<region>');
ATTACH 'ducklake:postgres:host=db.example.com dbname=lake user=analyst password=<password>' AS lake
(METADATA_SCHEMA '<metadata_schema>', READ_ONLY);
""")

rows = duckdb.sql("""
SELECT signal_type, delivery_batch_id, ingested_at, payload
FROM lake.<namespace>.<table>
WHERE ingested_at >= '2026-11-19'
""").pl()

records = records_from_table(rows.lazy())

Omit METADATA_SCHEMA if you did not set one on the destination. The DuckDB page covers the attach options.

pandas, through DuckDB​

To query with SQL and get a pandas DataFrame, run DuckDB in the same process. This reads bucket objects; the DuckDB page has the secrets and views for every destination, and all of its queries work here.

import duckdb

duckdb.sql("""
CREATE SECRET s2b_bucket (TYPE s3, PROVIDER credential_chain, REGION '<region>');
CREATE VIEW records AS
SELECT * FROM read_json('s3://<bucket>/<prefix>/s2b/stream=*/date=*/hour=*/*.ndjson.gz',
format = 'newline_delimited', hive_partitioning = true,
columns = {v: 'INTEGER', delivery_batch_id: 'VARCHAR', batch_id: 'VARCHAR',
received_at: 'TIMESTAMPTZ', signal: 'VARCHAR',
resource: 'JSON', scope: 'JSON', data: 'JSON'});
""")

events = duckdb.sql("""
SELECT data->>'$.event' AS event, count(*) AS events,
sum((data->>'$.cents')::INTEGER) / 100 AS revenue_usd
FROM records WHERE signal = 'event'
GROUP BY ALL ORDER BY events DESC
""").df()

.df() returns a pandas DataFrame and .pl() a Polars DataFrame.

Example queries​

These run against records from any of the Polars paths above. d is shorthand for the data column.

d = pl.col("data")

Records per signal:

(records
.group_by("signal")
.agg(records=pl.len())
.sort("records", descending=True)
.collect())

JSON events by name, with revenue from order.paid:

(records
.filter(pl.col("signal") == "event")
.group_by(event=d.struct.field("event"))
.agg(events=pl.len(), revenue_usd=d.struct.field("cents").sum() / 100)
.sort("events", descending=True)
.collect())
shape: (4, 3)
┌────────────┬────────┬─────────────┐
│ event ┆ events ┆ revenue_usd │
│ --- ┆ --- ┆ --- │
│ str ┆ u32 ┆ f64 │
╞════════════╪════════╪═════════════╡
│ page.view ┆ 177 ┆ 0.0 │
│ order.paid ┆ 53 ┆ 4161.0 │
│ email.sent ┆ 47 ┆ 0.0 │
│ signup ┆ 23 ┆ 0.0 │
└────────────┴────────┴─────────────┘

Polars sums missing values to 0, where SQL engines return NULL.

Log records by service and severity. service.name is one entry in the resource's attribute list, so filter the list by key and take the value:

service = (pl.col("resource").struct.field("attributes")
.list.filter(pl.element().struct.field("key") == "service.name")
.list.first()
.struct.field("value").struct.field("stringValue"))

(records
.filter(pl.col("signal") == "log")
.group_by(service=service, severity=d.struct.field("severityText"))
.agg(logs=pl.len())
.collect())

The five slowest spans. Timestamps are nanosecond strings; pl.Int64 holds Unix nanoseconds until the year 2262:

(records
.filter(pl.col("signal") == "span")
.select(
span=d.struct.field("name"),
trace_id=d.struct.field("traceId"),
duration_ms=(d.struct.field("endTimeUnixNano").cast(pl.Int64)
- d.struct.field("startTimeUnixNano").cast(pl.Int64)) / 1e6,
)
.sort("duration_ms", descending=True)
.head(5)
.collect())

Notes​

  • Bucket objects are gzip JSON, which is slower to scan than Parquet. Narrow the glob to the days you need, or write a day to Parquet once with records.sink_parquet(...) and query that. A table destination such as Apache Iceberg or DuckLake stores Parquet from the start.
  • str.json_decode requires a target type. Reuse RESOURCE and DATA so bucket and table paths give the same columns.
  • pl.scan_iceberg may print a PerformanceWarning suggesting the polars_iceberg package; without it, Polars plans the scan with PyIceberg.
  • In DuckDB 2.0, a relation from duckdb.sql() runs its query again on each fetch and no longer has fetchone(). Call .df(), .pl() or .fetchall() once and keep the result.
  • See Delivery format for every envelope field, and the querying overview for other engines.