Grafana
Grafana is a good fit for logs, spans and metrics: its panels are time series first, and its SQL data sources fill in the dashboard's time range and bucket size for you. Connect it to the engine that holds your Send2Bucket data.
| You have | Grafana data source |
|---|---|
| Records loaded into PostgreSQL or MySQL | Built-in PostgreSQL or MySQL |
| ClickHouse | ClickHouse plugin by Grafana Labs |
| Trino or Starburst | Trino community plugin |
| BigQuery | Google BigQuery plugin by Grafana Labs |
| Athena | Amazon Athena plugin by Grafana Labs |
| MotherDuck or a DuckDB file | MotherDuck's DuckDB plugin |
We ran the PostgreSQL data source in Grafana 13.3: provisioning, the time series and table queries on this page, through Grafana's query API. The plugins follow their own documentation and were not run by us.
PostgreSQL
Load records on a schedule as in PostgreSQL. Create a role that can only read, as in Metabase, then add the data source under Connections > Data sources > PostgreSQL, or provision it from a file in provisioning/datasources/:
apiVersion: 1
datasources:
- name: Send2Bucket
uid: s2b-postgres
type: grafana-postgresql-datasource
url: <postgres_host>:5432
user: grafana
jsonData:
database: <database>
sslmode: require
postgresVersion: 1500
secureJsonData:
password: $S2B_PG_PASSWORD
Grafana replaces $S2B_PG_PASSWORD with the environment variable of that name when it starts, so the password stays out of the file. Older Grafana releases call the type postgres.
Time series panels
$__timeFilter limits a query to the dashboard's time range, and $__timeGroupAlias buckets rows into a time column. Records per signal per minute, one series per signal:
SELECT $__timeGroupAlias(received_at, '1m', 0),
signal AS metric,
count(*) AS records
FROM s2b_records
WHERE $__timeFilter(received_at)
GROUP BY 1, 2
ORDER BY 1
Set the query's Format to Time series. A column named metric becomes the series name, and the 0 fills minutes with no records with zero instead of leaving gaps.
order.paid revenue per interval, with $__interval following the panel's width:
SELECT $__timeGroupAlias(received_at, $__interval, 0),
sum((data->>'cents')::numeric) / 100 AS revenue_usd
FROM s2b_records
WHERE $__timeFilter(received_at)
AND signal = 'event' AND data->>'event' = 'order.paid'
GROUP BY 1
ORDER BY 1
p95 span duration per minute:
SELECT $__timeGroupAlias(received_at, '1m'),
percentile_cont(0.95) WITHIN GROUP (
ORDER BY ((data->>'endTimeUnixNano')::numeric
- (data->>'startTimeUnixNano')::numeric) / 1e6) AS p95_ms
FROM s2b_records
WHERE $__timeFilter(received_at) AND signal = 'span'
GROUP BY 1
ORDER BY 1
received_at is when Send2Bucket accepted the record. To plot by when a span started, use to_timestamp((data->>'startTimeUnixNano')::numeric / 1e9) as the time column instead. That expression has no index, so keep $__timeFilter(received_at) in the WHERE clause as well.
Table panels
Set Format to Table for breakdowns. Logs by service and severity in the selected range:
SELECT jsonb_path_query_first(resource,
'$.attributes[*] ? (@.key == "service.name").value.stringValue') #>> '{}' AS service,
data->>'severityText' AS severity,
count(*) AS logs
FROM s2b_records
WHERE $__timeFilter(received_at) AND signal = 'log'
GROUP BY service, severity
ORDER BY logs DESC
The other questions come from PostgreSQL: Example queries with $__timeFilter(received_at) added.
MySQL
The MySQL data source has the same macros. Use the received_at column of the MySQL table, which holds UTC times, and set the data source's time zone to UTC.
ClickHouse, Trino, BigQuery and Athena
Install the plugin from Administration > Plugins and data > Plugins, or with grafana cli plugins install <plugin_id>, then add the data source with a read-only user or key:
| Data source | Plugin ID | Credentials | SQL |
|---|---|---|---|
| ClickHouse | grafana-clickhouse-datasource | User with SELECT on the tables | ClickHouse |
| Trino | trino-datasource | Trino user, over HTTPS | Trino |
| BigQuery | grafana-bigquery-datasource | Service account with BigQuery Data Viewer and Job User | BigQuery |
| Athena | grafana-athena-datasource | IAM keys or role allowed to run Athena queries and read the prefix | Athena |
Each plugin has its own time macros, such as $__timeFilter(column). Check the plugin's documentation for its exact names. Every panel refresh runs a query, which costs money on BigQuery and Athena: filter on the date partition column as well as the time column, and set dashboard refresh intervals of minutes, not seconds.
DuckDB and MotherDuck
MotherDuck publishes a DuckDB data source plugin (motherduck-duckdb-datasource) for local DuckDB files and MotherDuck databases. It is unsigned, so allow it with GF_PLUGINS_ALLOW_LOADING_UNSIGNED_PLUGINS=motherduck-duckdb-datasource, and it needs a glibc-based Grafana image. Point it at the views from MotherDuck or DuckDB. See MotherDuck's Grafana guide for installation.
Notes
- Grafana runs every panel's query on each dashboard load and refresh. A query over gzip objects in a bucket is slow for that, so chart a loaded table or a table destination.
- For alerting on Send2Bucket data, such as error log counts, use the same queries in Grafana alert rules. The query must return a time series or a single number.
- OTLP timestamps are nanosecond strings. Cast them before arithmetic, as above.