Skip to main content

Dremio

Dremio reads a bucket destination in place: add the bucket as an Amazon S3 source, promote the s2b folder as a JSON dataset, and query it with SQL. Dremio decompresses the gzip objects itself and turns each envelope into nested columns, so data.event and resource.attributes read like ordinary fields. Use it when your team already queries the lake through Dremio, or wants a semantic layer of views over Send2Bucket data.

note

We ran the S3-compatible source, the JSON dataset, the records view and the example queries on Dremio OSS 26.0.5. The Amazon S3 authentication settings and the Iceberg REST catalog section follow Dremio's documentation and were not run by us. The Iceberg REST catalog source is a Dremio Enterprise feature; Dremio OSS 26 does not offer it.

Connect​

Amazon S3​

In Dremio, choose Add Source and Amazon S3:

SettingValue
Names2b_bucket
AuthenticationEC2 Metadata (or EKS Pod Identity) with an IAM Role to Assume that can read the bucket, or AWS Access Key for a read-only IAM user
Encrypt connectionOn
Advanced Options, Allowlisted buckets<bucket>

The same source through the REST API, as an admin. Get a session token first (Dremio Enterprise also takes a personal access token):

curl -X POST "https://<dremio_host>:9047/apiv2/login" -H "Content-Type: application/json" \
-d '{"userName": "<admin_user>", "password": "<password>"}'
# {"token": "<token>", ...}

curl -X POST "https://<dremio_host>:9047/api/v3/catalog" \
-H "Authorization: Bearer <token>" -H "Content-Type: application/json" \
-d '{
"entityType": "source", "type": "S3", "name": "s2b_bucket",
"config": {
"credentialType": "EC2_METADATA",
"assumedRoleARN": "arn:aws:iam::<account_id>:role/<read_only_role>",
"secure": true,
"whitelistedBuckets": ["<bucket>"]
}
}'

S3-compatible storage​

Use the same Amazon S3 source type, with an access key and three more settings under Advanced Options:

SettingValue
Enable compatibility modeOn
Connection property fs.s3a.endpointThe endpoint host, without https://, such as <account_id>.r2.cloudflarestorage.com
Connection property fs.s3a.path.style.accesstrue
{
"entityType": "source", "type": "S3", "name": "s2b_bucket",
"config": {
"credentialType": "ACCESS_KEY",
"accessKey": "<access_key_id>",
"accessSecret": "<secret_access_key>",
"secure": true,
"compatibilityMode": true,
"whitelistedBuckets": ["<bucket>"],
"propertyList": [
{"name": "fs.s3a.endpoint", "value": "<endpoint_host>"},
{"name": "fs.s3a.path.style.access", "value": "true"}
]
}
}

Turn Encrypt connection (secure) off only for a plain-HTTP endpoint.

Promote the folder as a JSON dataset​

In the source, open <bucket> and <prefix>, hover over the s2b folder, choose Format Folder, and pick JSON. Through the API, post the folder's path to its catalog ID, dremio:/ followed by the path, URL-encoded:

curl -X POST "https://<dremio_host>:9047/api/v3/catalog/dremio%3A%2Fs2b_bucket%2F<bucket>%2F<prefix>%2Fs2b" \
-H "Authorization: Bearer <token>" -H "Content-Type: application/json" \
-d '{"entityType": "dataset", "type": "PHYSICAL_DATASET",
"path": ["s2b_bucket", "<bucket>", "<prefix>", "s2b"],
"format": {"type": "JSON"}}'

Dremio reads the .ndjson.gz objects without further settings. The dataset has the envelope's fields (v, delivery_batch_id, batch_id, received_at, signal, resource, scope, data) as columns, with resource, scope and data as structs, plus dir0, dir1 and dir2 for the stream=…, date=… and hour=… folders. Dremio skips folders whose names start with _, so the destination's test objects under s2b/_verify/ stay out.

Then run once:

SELECT * FROM s2b_bucket."<bucket>"."<prefix>".s2b;

Dremio builds the dataset's schema from the first files it samples. A field that only appears in other files, such as data.severityText when the sample held only events and spans, is unknown until a query reads every column; until then a query that names it fails with Column 'data.severityText' not found. SELECT * reads every column, records the new fields, and reruns itself. Run it again when your events gain new fields.

The records view​

Create a space and a view with the same columns as the other engine pages:

CREATE OR REPLACE VIEW s2b.records AS
SELECT SUBSTR(dir0, 8) AS "stream",
SUBSTR(dir1, 6) AS "date",
SUBSTR(dir2, 6) AS "hour",
signal,
TO_TIMESTAMP(received_at, 'YYYY-MM-DD"T"HH24:MI:SS.FFFFFF"Z"') AS received_at,
delivery_batch_id,
resource,
data
FROM s2b_bucket."<bucket>"."<prefix>".s2b
WHERE dir0 LIKE 'stream=%';
  • Create the space s2b first (Add Space in the UI, or {"entityType": "space", "name": "s2b"} posted to /api/v3/catalog).
  • stream, date and hour are reserved words in Dremio, so quote them.
  • received_at is a UTC TIMESTAMP with millisecond precision; Dremio drops the last three digits. A plain CAST fails on the T and Z.
  • Filter on "date" and "hour" to read fewer folders.

Apache Iceberg (REST catalog)​

Dremio Enterprise has an Iceberg REST Catalog source. Set Endpoint URI to <catalog_uri>, add warehouse = <warehouse> under the advanced properties, and either keep Use vended credentials on or add storage credentials (fs.s3a.aws.credentials.provider, fs.s3a.access.key, fs.s3a.secret.key). The namespace and table then appear under the source. payload is a JSON string, which Dremio parses with CONVERT_FROM(payload, 'JSON'); see the memory note under Notes. See Dremio: Iceberg REST Catalog.

Dremio OSS 26 has no REST catalog source, and it cannot read a REST catalog's table folder directly, because the folder has no version-hint.text. With Dremio OSS, use a bucket destination, or read the Iceberg table with DuckDB.

Example queries​

Nested fields need the table alias: write r.data.event, not data.event.

Records per signal:

SELECT signal, count(*) AS records
FROM s2b.records
GROUP BY signal
ORDER BY records DESC;

JSON events by name, with revenue from order.paid:

SELECT r.data.event AS event,
count(*) AS events,
sum(r.data.cents) / 100 AS revenue_usd
FROM s2b.records AS r
WHERE r.signal = 'event'
GROUP BY r.data.event
ORDER BY events DESC;
event | events | revenue_usd
page.view | 177 |
order.paid | 53 | 4161
email.sent | 47 |
signup | 23 |

Log records by service and severity. FLATTEN turns the resource's attribute list into one row per attribute, and the outer query keeps service.name:

SELECT t.attr."value".stringValue AS service, t.severity, count(*) AS logs
FROM (SELECT FLATTEN(r.resource.attributes) AS attr, r.data.severityText AS severity
FROM s2b.records AS r
WHERE r.signal = 'log') AS t
WHERE t.attr."key" = 'service.name'
GROUP BY t.attr."value".stringValue, t.severity;

The five slowest spans. Timestamps are nanosecond strings, so cast to BIGINT before subtracting:

SELECT r.data.name AS span,
r.data.traceId AS trace_id,
(CAST(r.data.endTimeUnixNano AS BIGINT) - CAST(r.data.startTimeUnixNano AS BIGINT)) / 1e6 AS duration_ms
FROM s2b.records AS r
WHERE r.signal = 'span'
ORDER BY duration_ms DESC
LIMIT 5;

Notes​

  • data is one struct whose fields are the union of every record's keys, so r.data.cents is NULL on records that lack it. Every field of your JSON events lands in the same struct as the OTLP fields; filter on signal first.
  • Every query reads the gzip objects again. For repeated queries, a raw reflection on s2b.records keeps an optimized copy that Dremio refreshes on a schedule; we did not run reflections.
  • Reading the lines as text and parsing them with CONVERT_FROM(line, 'JSON') ran out of memory on Dremio OSS 26 with a 1.5 GB direct-memory limit, even on a few hundred lines. The JSON dataset above does not.
  • Google Cloud Storage objects are not covered here. DuckLake tables are not readable from Dremio; use DuckDB.

See Delivery format for the envelope, and the Querying overview for other engines.