Adding a dataset#
A dataset is a folder with a dataset.toml and its small files, in the folder of its domain. This page
builds one in a temporary folder and loads it.
Put a dataset in the package at src/nbdemo/data/<domain>/<name>/. The domain is aero, med or rf.
The name is the Python name that users import.
Write the files and the manifest#
Files up to about 1 MB ship in the package. A larger file is not in the repository: its manifest entry has
a url and the file downloads on first use.
import hashlib
import tempfile
from pathlib import Path
folder = Path(tempfile.mkdtemp()) / "bench_voltage"
folder.mkdir()
(folder / "voltage.csv").write_text(
"time,volts\n2026-01-01T12:00:00Z,3.30\n2026-01-01T12:00:01Z,3.31\n"
)
63
just hash-files <folder> prints the [[files]] entries. This cell computes the same values:
data = (folder / "voltage.csv").read_bytes()
digest = hashlib.sha256(data).hexdigest()
(folder / "dataset.toml").write_text(f"""\
description = "Two readings of a 3.3 V rail"
license = "Apache-2.0"
source = "Written by hand for this page"
version = 1
[forms]
dataframe = "csv"
[unavailable]
native = "a CSV file has no object of its own; use .df"
[tabular]
index = "time"
[[files]]
name = "voltage.csv"
primary = true
sha256 = "{digest}"
size = {len(data)}
""")
378
The fields:
description,licenseandsourceare required. A license that asks for credit, such asCC-BY-4.0, also needs anattribution.versiongoes up when a file’s content changes. Downloads are cached by version.[[files]]lists every file, with exactly oneprimary.pathis that file.[forms]names a loader fornativeanddataframe. The loaders arewfdb,sigmfandcsv. Leave a form out if the format has none, and say why under[unavailable].
Load it#
Dataset.from_dir reads any folder with a manifest and checks it, so you can try it before adding it to
the package.
from nbdemo.data import Dataset
dataset = Dataset.from_dir(folder, domain="aero")
dataset.df
| volts | |
|---|---|
| time | |
| 2026-01-01 12:00:00+00:00 | 3.30 |
| 2026-01-01 12:00:01+00:00 | 3.31 |
verify compares the files with their manifest entries, as the tests do for every dataset:
dataset.verify() # an empty list: every file matches
[]
Generated data#
Data that a script makes goes with the script. Put the script in gen/, add a [generated] table to the
manifest with its script, seed and command, and check in the output. gen/aero_flight_dc.py makes
aero/flight_dc, and a test reruns it to confirm that it reproduces the checked-in file:
import nbdemo.data as nd
nd.aero.flight_dc.manifest["generated"]
{'script': 'gen/aero_flight_dc.py', 'seed': 42, 'command': 'just gen-flight'}