Skip to main content

Deepnote

Deepnote notebooks mix Python blocks with SQL blocks. A SQL block runs either on DuckDB inside the notebook ("DataFrame SQL") or on a warehouse that the workspace has connected as an integration. Pick the path by where your data already is.

You haveUse
No warehouse yetDuckDB in a Python block, then DataFrame SQL blocks
Trino, Athena, BigQuery, ClickHouse, Redshift or PostgreSQL over your dataA warehouse integration
A MotherDuck accountMotherDuck from a Python block
note

This page follows Deepnote's documentation. The steps in Deepnote were not run by us. We ran the DuckDB SQL on this page in DuckDB 2.0.0 against a local copy of our sample objects; the MotherDuck snippet follows MotherDuck's documentation.

Credentials​

Keys go in an Environment variables integration, not in the notebook:

  1. In the project's right-hand sidebar, open Integrations and add an Environment variables integration with S2B_KEY_ID and S2B_SECRET, holding a read-only storage key.
  2. Connect the integration to the project. Python blocks read the values with os.environ.

Deepnote stores the values encrypted and mounts them only into connected projects. Terminal sessions do not see them unless you run source <(/compute-helpers/code/print-integration-env-vars.py) in the terminal.

DuckDB in a Python block​

DuckDB runs inside the notebook and reads your bucket or table directly. Use the Python cell from Jupyter unchanged: it creates the records view from os.environ["S2B_KEY_ID"] and os.environ["S2B_SECRET"], and returns a pandas DataFrame named events. For a table destination, replace the view with the ATTACH and view for your catalog from DuckDB.

Run !pip install duckdb first if the project's environment does not have it, or a newer version than it ships.

Then add a SQL block, set its data source to DataFrame SQL, and query the DataFrame by its variable name:

SELECT event, events, revenue_usd
FROM events
WHERE revenue_usd IS NOT NULL

DataFrame SQL runs on DuckDB, so the example queries work in it too once a DataFrame holds the rows they read. A SQL block stores its full result in a DataFrame variable you name, which later Python blocks and charts use.

Reading a mounted bucket​

An Amazon S3 integration mounts a bucket into the project under /datasets/<integration_name>, with an access key or an IAM role; a Google Cloud Storage integration mounts a bucket with a service account key. DataFrame SQL can then read the objects by path, without a Python block:

SELECT data->>'$.event' AS event, count(*) AS events
FROM read_json(
'/datasets/<integration_name>/<prefix>/s2b/stream=*/date=2026-11-19/hour=*/*.ndjson.gz',
format = 'newline_delimited', hive_partitioning = true,
columns = {v: 'INTEGER', delivery_batch_id: 'VARCHAR', batch_id: 'VARCHAR',
received_at: 'TIMESTAMPTZ', signal: 'VARCHAR',
resource: 'JSON', scope: 'JSON', data: 'JSON'})
WHERE signal = 'event'
GROUP BY ALL

Name a single day or hour in the path: every object the glob matches is read through the mount.

Warehouse integrations​

If your team already queries Send2Bucket data through a warehouse, add it as an integration and query the same tables or views:

IntegrationSet up the tables with
Amazon AthenaAmazon Athena. The integration needs the region, an S3 output location for results, and an access key.
TrinoTrino
Google BigQueryBigQuery
ClickHouseClickHouse
Amazon RedshiftAmazon Redshift
PostgreSQLPostgreSQL

Integrations are created at the workspace level (project Settings, then Integrations) and connected to each project that uses them, so analysts do not hold storage keys. Give the integration a database user that can only read. In a SQL block, pick the integration as the data source and use the example queries from the engine's page.

Deepnote also lists Snowflake, Databricks and other warehouses as integrations; the same approach applies if your data is in one of them.

Query caching​

On Team and Enterprise plans, SQL blocks connected to an integration can cache results for a project-wide period (1 hour to 1 week, 1 day by default). A cached block does not show deliveries that arrived after the cached run. Use Run without cache on the block, or Run notebook without cache, to read the latest data. Scheduled runs always query the warehouse. DataFrame SQL blocks are never cached.

MotherDuck​

Deepnote does not list a MotherDuck integration. Connect from a Python block instead: add motherduck_token (a read-scoped token) to the Environment variables integration, then:

import duckdb

con = duckdb.connect("md:s2b")
events = con.sql("SELECT signal, count(*) AS records FROM records GROUP BY ALL").df()

This reads the records view you create in MotherDuck with MotherDuck. DataFrame SQL blocks can then query events.

Notes​

  • DuckDB in a Python block and DataFrame SQL hold results in the project machine's memory. Filter by date (bucket objects) or ingested_at (tables) before pulling rows into a DataFrame, or pick a larger machine.
  • Each run of a DuckDB block reads the bucket again and sees the latest deliveries. Warehouse integrations do too, unless a cached result is served.