Apache Spark
Spark reads the gzip NDJSON objects that Amazon S3, Google Cloud Storage and S3-compatible destinations write with its built-in JSON reader, and reads Apache Iceberg, Amazon S3 Tables and Supabase Analytics Buckets destinations through the Iceberg Spark runtime. Use it when you already run Spark, on your own cluster or on EMR, Dataproc or AWS Glue, and want Send2Bucket data in the same jobs as your other sources. For Databricks, see Databricks. For the record layout, see Delivery format.
We ran the S3-compatible and Iceberg REST sections, the metastore table and every example query on Spark 4.0.2, and the Spark 3.5 queries on Spark 3.5.7, against our sample data. The Google Cloud Storage, Amazon S3 Tables and Supabase Analytics Buckets settings follow the Google Cloud, AWS and Supabase documentation; we have not run them against those services.
Connect to bucket objects
Each section ends with a temporary view named records, with signal, received_at, resource and data columns. The example queries run against that view.
Amazon S3
Spark reads s3a:// paths through the hadoop-aws package. Its version must match the Hadoop version your Spark build ships with: 3.4.1 for Spark 4.0, 3.3.4 for Spark 3.5. In PySpark:
from pyspark.sql import SparkSession
spark = (SparkSession.builder
.config("spark.jars.packages", "org.apache.hadoop:hadoop-aws:3.4.1")
.config("spark.hadoop.fs.s3a.endpoint.region", "<region>")
.getOrCreate())
Without keys in the configuration, S3A uses the standard AWS credential sources: environment variables and the instance or task role. On EMR, Dataproc and AWS Glue the S3 connector is already installed, so skip spark.jars.packages; EMR and Glue also read s3:// paths.
Then define the view in Spark SQL (spark.sql("…") in PySpark):
CREATE TEMPORARY VIEW records (
v INT, delivery_batch_id STRING, batch_id STRING, received_at TIMESTAMP,
signal STRING, resource VARIANT, scope VARIANT, data VARIANT)
USING json
OPTIONS (
path 's3a://<bucket>/<prefix>/s2b/stream=*/date=*/hour=*/*.ndjson.gz',
basePath 's3a://<bucket>/<prefix>/s2b/');
basePathturns thestream=,date=andhour=folders intostream STRING,date DATEandhour INTcolumns. Without it, a path with wildcards has no partition columns.WHERE date = '2026-11-19'then reads only that day's folders.- Spark decompresses the objects because their names end in
.gz. - The column list keeps types stable. Without it, Spark infers a schema from the data and turns
datainto a struct of whichever event fields it saw. VARIANTneeds Spark 4.0. On Spark 3.5, declareresource,scopeanddataasSTRINGand use the Spark 3.5 queries.stream=*keeps the destination's test objects unders2b/_verify/out of the view.
The same view from the PySpark DataFrame API:
base = "s3a://<bucket>/<prefix>/s2b/"
records = (spark.read
.schema("v INT, delivery_batch_id STRING, batch_id STRING, received_at TIMESTAMP, "
"signal STRING, resource VARIANT, scope VARIANT, data VARIANT")
.option("basePath", base)
.json(base + "stream=*/date=*/hour=*/*.ndjson.gz"))
records.createOrReplaceTempView("records")
Spark lists the objects when it first plans a query on the view. In a long-running session, REFRESH TABLE records picks up objects delivered since.
A table in the metastore
To share the definition through a Hive metastore or AWS Glue Data Catalog, create a partitioned table over the same folder:
CREATE TABLE s2b_records (
v INT, delivery_batch_id STRING, batch_id STRING, received_at TIMESTAMP,
signal STRING, resource STRING, scope STRING, data STRING,
stream STRING, date DATE, hour INT)
USING json
PARTITIONED BY (stream, date, hour)
LOCATION 's3a://<bucket>/<prefix>/s2b/';
MSCK REPAIR TABLE s2b_records;
MSCK REPAIR TABLE registers the hour folders that exist now. Run it again every hour, or after each delivery, so new hours show up. The columns are STRING here so that any engine sharing the metastore can read the table; parse them with parse_json on Spark 4.0 or with the Spark 3.5 functions.
Google Cloud Storage
On Dataproc, gs:// paths work with the cluster's service account. Elsewhere, add the Cloud Storage connector and give it a service account that can read the bucket. Then create the view as for Amazon S3, with gs://<bucket>/<prefix>/s2b/stream=*/date=*/hour=*/*.ndjson.gz and basePath 'gs://<bucket>/<prefix>/s2b/'.
S3-compatible storage
Add the endpoint and, for most self-hosted stores, path-style addressing:
spark = (SparkSession.builder
.config("spark.jars.packages", "org.apache.hadoop:hadoop-aws:3.4.1")
.config("spark.hadoop.fs.s3a.endpoint", "https://<endpoint>")
.config("spark.hadoop.fs.s3a.endpoint.region", "<region>")
.config("spark.hadoop.fs.s3a.path.style.access", "true")
.config("spark.hadoop.fs.s3a.access.key", "<access_key_id>")
.config("spark.hadoop.fs.s3a.secret.key", "<secret_access_key>")
.getOrCreate())
For Cloudflare R2, the endpoint is https://<account_id>.r2.cloudflarestorage.com and the region auto. For an endpoint on plain HTTP, also set spark.hadoop.fs.s3a.connection.ssl.enabled to false. Then create the view as for Amazon S3.
Connect to an Iceberg table
Add the Iceberg runtime for your Spark and Scala version, and iceberg-aws-bundle for S3 storage, then define a catalog with the same catalog URI, warehouse and storage key you gave Send2Bucket:
spark = (SparkSession.builder
.config("spark.jars.packages",
"org.apache.iceberg:iceberg-spark-runtime-4.0_2.13:1.10.1,"
"org.apache.iceberg:iceberg-aws-bundle:1.10.1")
.config("spark.sql.extensions",
"org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions")
.config("spark.sql.catalog.lake", "org.apache.iceberg.spark.SparkCatalog")
.config("spark.sql.catalog.lake.type", "rest")
.config("spark.sql.catalog.lake.uri", "<catalog_uri>")
.config("spark.sql.catalog.lake.warehouse", "<warehouse>")
.config("spark.sql.catalog.lake.token", "<catalog_token>")
.config("spark.sql.catalog.lake.io-impl", "org.apache.iceberg.aws.s3.S3FileIO")
.config("spark.sql.catalog.lake.client.region", "<region>")
.config("spark.sql.catalog.lake.s3.access-key-id", "<access_key_id>")
.config("spark.sql.catalog.lake.s3.secret-access-key", "<secret_access_key>")
.getOrCreate())
- On Spark 3.5, use
iceberg-spark-runtime-3.5_2.12(or_2.13for a Scala 2.13 build). lakeis the catalog name in SQL. The table islake.<namespace>.<table>.- A read-only token and storage key are enough. Leave out the two
s3.keys to use the default AWS credential chain. - If the table's data files are in S3-compatible storage, add
spark.sql.catalog.lake.s3.endpoint(https://<endpoint>) andspark.sql.catalog.lake.s3.path-style-access(true).
The table has the four table columns. Parse payload once to get the same view as for objects:
CREATE TEMPORARY VIEW records AS
SELECT signal_type AS signal, ingested_at AS received_at, delivery_batch_id,
variant_get(envelope, '$.resource') AS resource,
variant_get(envelope, '$.data') AS data
FROM (SELECT *, parse_json(payload) AS envelope FROM lake.<namespace>.<table>);
On Spark 3.5, query payload with get_json_object(payload, '$.data.event') and the other Spark 3.5 functions.
Amazon S3 Tables
S3 Tables has an Iceberg REST endpoint signed with SigV4. Replace the catalog settings with:
.config("spark.sql.catalog.lake.type", "rest")
.config("spark.sql.catalog.lake.uri", "https://s3tables.<region>.amazonaws.com/iceberg")
.config("spark.sql.catalog.lake.warehouse", "arn:aws:s3tables:<region>:<account_id>:bucket/<table_bucket>")
.config("spark.sql.catalog.lake.rest.sigv4-enabled", "true")
.config("spark.sql.catalog.lake.rest.signing-name", "s3tables")
.config("spark.sql.catalog.lake.rest.signing-region", "<region>")
.config("spark.sql.catalog.lake.io-impl", "org.apache.iceberg.aws.s3.S3FileIO")
Credentials come from the default AWS chain. The identity needs s3tables:GetTableBucket, s3tables:GetNamespace, s3tables:GetTable, s3tables:GetTableMetadataLocation and s3tables:GetTableData on the table bucket.
Supabase Analytics Buckets
.config("spark.sql.catalog.lake.type", "rest")
.config("spark.sql.catalog.lake.uri", "https://<project_ref>.supabase.co/storage/v1/iceberg")
.config("spark.sql.catalog.lake.warehouse", "<analytics_bucket>")
.config("spark.sql.catalog.lake.token", "<service_key>")
.config("spark.sql.catalog.lake.io-impl", "org.apache.iceberg.aws.s3.S3FileIO")
.config("spark.sql.catalog.lake.s3.endpoint", "https://<project_ref>.supabase.co/storage/v1/s3")
.config("spark.sql.catalog.lake.s3.path-style-access", "true")
.config("spark.sql.catalog.lake.s3.access-key-id", "<access_key_id>")
.config("spark.sql.catalog.lake.s3.secret-access-key", "<secret_access_key>")
.config("spark.sql.catalog.lake.s3.remote-signing-enabled", "false")
.config("spark.sql.catalog.lake.client.region", "<region>")
Example queries
These run on Spark 4.0 against the records view from any section above. Spark 4.0 reads VARIANT fields with variant_get(column, path, type); the column:field shorthand is Databricks syntax and fails in Apache Spark.
Records per signal:
SELECT signal, count(*) AS records
FROM records
GROUP BY ALL
ORDER BY records DESC;
JSON events by name, with revenue from order.paid:
SELECT variant_get(data, '$.event', 'string') AS event,
count(*) AS events,
sum(variant_get(data, '$.cents', 'int')) / 100 AS revenue_usd
FROM records
WHERE signal = 'event'
GROUP BY ALL
ORDER BY events DESC;
+----------+------+-----------+
| event|events|revenue_usd|
+----------+------+-----------+
| page.view| 177| NULL|
|order.paid| 53| 4161.0|
|email.sent| 47| NULL|
| signup| 23| NULL|
+----------+------+-----------+
Log records by service and severity. service.name is one entry in the resource's attribute list, so cast the list to an array and filter it:
SELECT variant_get(
filter(variant_get(resource, '$.attributes', 'array<variant>'),
a -> variant_get(a, '$.key', 'string') = 'service.name')[0],
'$.value.stringValue', 'string') AS service,
variant_get(data, '$.severityText', 'string') AS severity,
count(*) AS logs
FROM records
WHERE signal = 'log'
GROUP BY ALL;
The five slowest spans. The timestamps are nanosecond strings; variant_get casts them to bigint, which holds nanoseconds until the year 2262:
SELECT variant_get(data, '$.name', 'string') AS span,
variant_get(data, '$.traceId', 'string') AS trace_id,
(variant_get(data, '$.endTimeUnixNano', 'bigint')
- variant_get(data, '$.startTimeUnixNano', 'bigint')) / 1e6 AS duration_ms
FROM records
WHERE signal = 'span'
ORDER BY duration_ms DESC
LIMIT 5;
Metrics by name and type. A metric's data has one key per type, which variant_explode turns into rows:
SELECT variant_get(r.data, '$.name', 'string') AS metric, e.key AS type, count(*) AS metrics
FROM records AS r, LATERAL variant_explode(r.data) AS e
WHERE r.signal = 'metric'
AND e.key IN ('gauge', 'sum', 'histogram', 'exponentialHistogram', 'summary')
GROUP BY ALL;
Spark 3.5
Spark 3.5 has no VARIANT type. Declare the JSON columns as STRING, which keeps each object as its JSON text:
CREATE TEMPORARY VIEW records (
v INT, delivery_batch_id STRING, batch_id STRING, received_at TIMESTAMP,
signal STRING, resource STRING, scope STRING, data STRING)
USING json
OPTIONS (
path 's3a://<bucket>/<prefix>/s2b/stream=*/date=*/hour=*/*.ndjson.gz',
basePath 's3a://<bucket>/<prefix>/s2b/');
Then read fields with get_json_object, and parse the attribute list with from_json:
SELECT signal, count(*) AS records
FROM records
GROUP BY signal
ORDER BY records DESC;
SELECT get_json_object(data, '$.event') AS event,
count(*) AS events,
sum(CAST(get_json_object(data, '$.cents') AS INT)) / 100 AS revenue_usd
FROM records
WHERE signal = 'event'
GROUP BY 1
ORDER BY events DESC;
SELECT filter(from_json(resource, 'attributes ARRAY<STRUCT<key: STRING, value: STRUCT<stringValue: STRING>>>').attributes,
a -> a.key = 'service.name')[0].value.stringValue AS service,
get_json_object(data, '$.severityText') AS severity,
count(*) AS logs
FROM records
WHERE signal = 'log'
GROUP BY 1, 2;
SELECT get_json_object(data, '$.name') AS span,
get_json_object(data, '$.traceId') AS trace_id,
(CAST(get_json_object(data, '$.endTimeUnixNano') AS BIGINT)
- CAST(get_json_object(data, '$.startTimeUnixNano') AS BIGINT)) / 1e6 AS duration_ms
FROM records
WHERE signal = 'span'
ORDER BY duration_ms DESC
LIMIT 5;
These queries also run on Spark 4.0 against the STRING columns of the metastore table. For an Iceberg table, read from payload instead: get_json_object(payload, '$.data.event'), and get_json_object(payload, '$.resource') in place of resource.
Notes
- Package versions.
hadoop-awsmust match Spark's Hadoop version, andiceberg-spark-runtime-<spark>_<scala>must match Spark's minor version and Scala version.hadoop-aws3.4.1 pulls in the AWS SDK bundle, about 550 MB, on first use; setspark.jars.ivyor bake the jars into your image to avoid downloading it on every new machine. - Gzip and parallelism. A gzip object cannot be split, so Spark reads each object in one task. Objects hold at most 16 MiB of uncompressed records, so this rarely matters, but a query over a month of a busy stream runs many small tasks. For repeated heavy queries, write a day at a time to Parquet or use a table destination such as Apache Iceberg.
- Time zone.
received_atandingested_atareTIMESTAMPvalues shown in the session time zone. Setspark.sql.session.timeZonetoUTCto see them as Send2Bucket wrote them. - New data. The temporary view lists objects when it is first queried, and the metastore table knows only the partitions
MSCK REPAIR TABLEregistered. An Iceberg catalog table shows each new append on the next query. - DuckLake. Spark has no DuckLake reader. Use DuckDB for a DuckLake destination.
- For the same questions in other engines, see the overview.