Skip to main content

Rill

Rill loads data into an embedded DuckDB (or ClickHouse) database and serves dashboards from it. For Send2Bucket, a Rill model reads the gzip NDJSON objects from your bucket, keeps a typed copy, and adds new objects on a schedule. A metrics view on top defines the measures people explore.

Use Rill when you want interactive dashboards over bucket objects without running a warehouse. For a table destination, a query engine reads the table in place.

note

We ran the S3 connector, the incremental model, the metrics view and its dashboard with Rill Developer 0.92.1 against an S3-compatible bucket, and queried them with rill query. The Google Cloud Storage connector and Rill Cloud deployment follow Rill's documentation and were not run by us.

Install​

curl https://rill.sh | sh
rill init s2b-rill --olap duckdb

rill init creates rill.yaml and a connectors/duckdb.yaml for the embedded DuckDB database.

Connect the bucket​

Add connectors/s3.yaml:

type: connector
driver: s3
aws_access_key_id: "{{ .env.aws_access_key_id }}"
aws_secret_access_key: "{{ .env.aws_secret_access_key }}"
region: <region>

Put a read-only key in the project's .env file, which rill init lists in .gitignore:

aws_access_key_id=<access_key_id>
aws_secret_access_key=<secret_access_key>

For S3-compatible storage, add endpoint: https://<endpoint> to the connector. For Google Cloud Storage, use driver: gcs with an HMAC key_id and secret, and gs:// paths below.

Load records with a model​

models/s2b_records.yaml treats each delivered object as one partition. Rill lists the objects that match the glob, runs the SQL once per object, and on each refresh runs it only for objects it has not loaded yet. Send2Bucket never changes an object after writing it, so each one is loaded exactly once.

type: model
connector: duckdb
create_secrets_from_connectors: s3
materialize: true
incremental: true

partitions:
glob:
connector: s3
path: s3://<bucket>/<prefix>/s2b/stream=*/date=*/hour=*/*.ndjson.gz

sql: |
SELECT received_at, signal, delivery_batch_id, stream, date,
data->>'$.event' AS event,
(data->>'$.cents')::INTEGER AS cents,
data->>'$.severityText' AS severity,
(SELECT a->>'$.value.stringValue'
FROM unnest(json_extract(resource, '$.attributes[*]')) AS t(a)
WHERE a->>'$.key' = 'service.name') AS service,
data->>'$.name' AS span_name,
((data->>'$.endTimeUnixNano')::HUGEINT
- (data->>'$.startTimeUnixNano')::HUGEINT) / 1e6 AS duration_ms
FROM read_json('{{ .partition.uri }}',
format = 'newline_delimited', hive_partitioning = true,
columns = {v: 'INTEGER', delivery_batch_id: 'VARCHAR', batch_id: 'VARCHAR',
received_at: 'TIMESTAMPTZ', signal: 'VARCHAR',
resource: 'JSON', scope: 'JSON', data: 'JSON'})

refresh:
cron: "*/15 * * * *"
  • create_secrets_from_connectors hands the S3 connector's key to DuckDB, so read_json can read the object.
  • The stream=* glob skips the test objects destination checks write to <prefix>/s2b/_verify/.
  • The SELECT turns the envelope into flat columns, one per field you want to filter or measure. It is the same SQL as the DuckDB example queries, applied per record. Add columns for your own event fields.
  • refresh.cron picks up new objects every 15 minutes in a deployed project. Rill Developer skips cron refreshes unless you add run_in_dev: true under refresh.

A first load reads every object under the prefix. To keep local development fast, limit the glob in development with a dev: block that overrides partitions with a single day's path.

Define a metrics view​

metrics/s2b_records.yaml:

version: 1
type: metrics_view
display_name: Send2Bucket records
model: s2b_records
timeseries: received_at
smallest_time_grain: minute

dimensions:
- name: signal
column: signal
- name: event
column: event
- name: service
column: service
- name: severity
column: severity
- name: span_name
column: span_name

measures:
- name: records
expression: COUNT(*)
format_preset: humanize
- name: revenue_usd
display_name: Revenue
expression: SUM(cents) FILTER (WHERE event = 'order.paid') / 100
format_preset: currency_usd
- name: p95_span_ms
display_name: p95 span duration (ms)
expression: QUANTILE_CONT(duration_ms, 0.95) FILTER (WHERE signal = 'span')
valid_percent_of_total: false

explore:
display_name: Send2Bucket

The explore block creates a dashboard for the metrics view, with every dimension and measure.

Run it​

rill start s2b-rill

Rill loads the model, then opens the dashboard in your browser. rill query runs SQL against a running project, for example to check the load:

rill query --local --sql "SELECT signal, count(*) AS records FROM s2b_records GROUP BY ALL"

On the sample data, the model has 300 events, 40 spans, 20 logs and 20 metrics. The revenue_usd measure for order.paid is 4161, and every log has service checkout and severity Info.

Notes​

  • Rill stores a copy of the loaded columns in its DuckDB database. Records are as fresh as the last refresh, and the copy grows with your data. Select only the columns you need.
  • Deploy with rill deploy to run the project in Rill Cloud. By default it pushes your local .env to the cloud project as well (--push-env), so the deployed project needs a key that can read the bucket from Rill Cloud. Change it later with rill env set or rill env push.
  • Rill's own documentation covers S3 and other connectors, and incremental models.