Skip to main content

Grafana

Grafana is a good fit for logs, spans and metrics: its panels are time series first, and its SQL data sources fill in the dashboard's time range and bucket size for you. Connect it to the engine that holds your Send2Bucket data.

You haveGrafana data source
Records loaded into PostgreSQL or MySQLBuilt-in PostgreSQL or MySQL
ClickHouseClickHouse plugin by Grafana Labs
Trino or StarburstTrino community plugin
BigQueryGoogle BigQuery plugin by Grafana Labs
AthenaAmazon Athena plugin by Grafana Labs
MotherDuck or a DuckDB fileMotherDuck's DuckDB plugin
note

We ran the PostgreSQL data source in Grafana 13.3: provisioning, the time series and table queries on this page, through Grafana's query API. The plugins follow their own documentation and were not run by us.

PostgreSQL​

Load records on a schedule as in PostgreSQL. Create a role that can only read, as in Metabase, then add the data source under Connections > Data sources > PostgreSQL, or provision it from a file in provisioning/datasources/:

apiVersion: 1
datasources:
- name: Send2Bucket
uid: s2b-postgres
type: grafana-postgresql-datasource
url: <postgres_host>:5432
user: grafana
jsonData:
database: <database>
sslmode: require
postgresVersion: 1500
secureJsonData:
password: $S2B_PG_PASSWORD

Grafana replaces $S2B_PG_PASSWORD with the environment variable of that name when it starts, so the password stays out of the file. Older Grafana releases call the type postgres.

Time series panels​

$__timeFilter limits a query to the dashboard's time range, and $__timeGroupAlias buckets rows into a time column. Records per signal per minute, one series per signal:

SELECT $__timeGroupAlias(received_at, '1m', 0),
signal AS metric,
count(*) AS records
FROM s2b_records
WHERE $__timeFilter(received_at)
GROUP BY 1, 2
ORDER BY 1

Set the query's Format to Time series. A column named metric becomes the series name, and the 0 fills minutes with no records with zero instead of leaving gaps.

order.paid revenue per interval, with $__interval following the panel's width:

SELECT $__timeGroupAlias(received_at, $__interval, 0),
sum((data->>'cents')::numeric) / 100 AS revenue_usd
FROM s2b_records
WHERE $__timeFilter(received_at)
AND signal = 'event' AND data->>'event' = 'order.paid'
GROUP BY 1
ORDER BY 1

p95 span duration per minute:

SELECT $__timeGroupAlias(received_at, '1m'),
percentile_cont(0.95) WITHIN GROUP (
ORDER BY ((data->>'endTimeUnixNano')::numeric
- (data->>'startTimeUnixNano')::numeric) / 1e6) AS p95_ms
FROM s2b_records
WHERE $__timeFilter(received_at) AND signal = 'span'
GROUP BY 1
ORDER BY 1

received_at is when Send2Bucket accepted the record. To plot by when a span started, use to_timestamp((data->>'startTimeUnixNano')::numeric / 1e9) as the time column instead. That expression has no index, so keep $__timeFilter(received_at) in the WHERE clause as well.

Table panels​

Set Format to Table for breakdowns. Logs by service and severity in the selected range:

SELECT jsonb_path_query_first(resource,
'$.attributes[*] ? (@.key == "service.name").value.stringValue') #>> '{}' AS service,
data->>'severityText' AS severity,
count(*) AS logs
FROM s2b_records
WHERE $__timeFilter(received_at) AND signal = 'log'
GROUP BY service, severity
ORDER BY logs DESC

The other questions come from PostgreSQL: Example queries with $__timeFilter(received_at) added.

MySQL​

The MySQL data source has the same macros. Use the received_at column of the MySQL table, which holds UTC times, and set the data source's time zone to UTC.

ClickHouse, Trino, BigQuery and Athena​

Install the plugin from Administration > Plugins and data > Plugins, or with grafana cli plugins install <plugin_id>, then add the data source with a read-only user or key:

Data sourcePlugin IDCredentialsSQL
ClickHousegrafana-clickhouse-datasourceUser with SELECT on the tablesClickHouse
Trinotrino-datasourceTrino user, over HTTPSTrino
BigQuerygrafana-bigquery-datasourceService account with BigQuery Data Viewer and Job UserBigQuery
Athenagrafana-athena-datasourceIAM keys or role allowed to run Athena queries and read the prefixAthena

Each plugin has its own time macros, such as $__timeFilter(column). Check the plugin's documentation for its exact names. Every panel refresh runs a query, which costs money on BigQuery and Athena: filter on the date partition column as well as the time column, and set dashboard refresh intervals of minutes, not seconds.

DuckDB and MotherDuck​

MotherDuck publishes a DuckDB data source plugin (motherduck-duckdb-datasource) for local DuckDB files and MotherDuck databases. It is unsigned, so allow it with GF_PLUGINS_ALLOW_LOADING_UNSIGNED_PLUGINS=motherduck-duckdb-datasource, and it needs a glibc-based Grafana image. Point it at the views from MotherDuck or DuckDB. See MotherDuck's Grafana guide for installation.

Notes​

  • Grafana runs every panel's query on each dashboard load and refresh. A query over gzip objects in a bucket is slow for that, so chart a loaded table or a table destination.
  • For alerting on Send2Bucket data, such as error log counts, use the same queries in Grafana alert rules. The query must return a time series or a single number.
  • OTLP timestamps are nanosecond strings. Cast them before arithmetic, as above.