nbdemo.data API#

Sample datasets for Nominal’s open-source projects.

Datasets are grouped by domain: aero, med and rf. Import one from its domain; each has a path, and where it makes sense a native object and a DataFrame:

from nbdemo.data.med import ecg_mitbih_119

ecg_mitbih_119.path    # a Path, to pass to your own loader
ecg_mitbih_119.native  # a wfdb.Record
ecg_mitbih_119.df      # a pandas.DataFrame
catalog(domain: str | None = None) → list[Dataset]#

Return every dataset, or the datasets of one domain.

Parameters:

domain

A domain name such as "rf"; None for all domains.

Returns:

The datasets, ordered by domain and name. Reads manifests only; nothing is downloaded.

get(dataset_id: str) → Dataset#

Return a dataset by its "<domain>/<name>", for when the name is only known at run time.

Parameters:

dataset_id

For example "med/ecg_mitbih_119".

Raises:

AttributeError – There is no such dataset.

domains() → list[str]#

Return the names of the domains.

cache_dir() → Path#

Return the directory that fetched files are saved in.

It is $NBDEMO_CACHE if set, otherwise nbdemo in $XDG_CACHE_HOME (~/.cache by default).

Returns:

The cache directory. It may not exist yet.

Domains#

Each domain module resolves dataset names when you import them, so there is no fixed list of attributes. dir() of a domain lists its datasets; the catalog shows them all.

The aero datasets. Import one by name; see nbdemo.data.catalog() for the list.

The med datasets. Import one by name; see nbdemo.data.catalog() for the list.

The rf datasets. Import one by name; see nbdemo.data.catalog() for the list.

Dataset#

class Dataset(domain: str, root: Path)#

A sample dataset: its files, and the forms they can be read in.

Get one by importing it from its domain, which reads only its manifest:

from nbdemo.data.med import ecg_mitbih_119

ecg_mitbih_119.path    # a Path, to pass to your own loader
ecg_mitbih_119.native  # a wfdb.Record
ecg_mitbih_119.df      # a pandas.DataFrame

Nothing is downloaded or parsed until you use path, paths, native or df. Files that do not ship in the package are downloaded on first use and kept in cache_dir().

Parameters:
  • domain

    The domain the dataset is in, such as "med".

  • root

    The folder holding dataset.toml and the files that ship in the package.

classmethod from_dir(root: str | Path, domain: str = 'local') → Dataset#

Read a dataset from any folder that holds a dataset.toml.

Use it to try a manifest before adding the dataset to the package.

Parameters:
  • root

    The dataset’s folder; its name is the dataset’s name.

  • domain

    A label for the dataset’s domain.

Raises:

ManifestError – The manifest is missing a field or has an invalid one.

property id: str#

The dataset’s <domain>/<name>.

property size: int#

The total size of the dataset’s files, in bytes.

property is_local: bool#

Whether every file is present, so using the dataset downloads nothing.

property forms: tuple[str, ...]#

The forms this dataset has. path is always one; native and df are where they apply.

property paths: dict[str, Path]#

Every file of the dataset, by name, downloading any that are missing.

The files are in one folder, so a loader can find companions such as a .hea beside its .dat.

property path: Path#

The path of the dataset’s main file, downloading the dataset’s files if needed.

The other files are beside it; see paths.

property native: Any#

The dataset as its format’s own Python object, such as a wfdb.Record.

Raises:
property df: Any#

The dataset as a pandas.DataFrame.

Raises:
verify() → list[str]#

Check the files that are present against the manifest.

Returns:

A description of each problem found; empty if every present file matches.

class FileSpec(name: str, sha256: str, size: int, url: str | None = None)#

One file of a dataset, as the manifest lists it.

url: str | None = None#

Where to download the file from. None for a file that ships in the package.

Errors#

exception SamplesError#

Base class for errors raised by nbdemo.

exception ManifestError#

A dataset.toml is missing a field or has an invalid one.

exception ChecksumError#

A downloaded file does not match the sha256 in its manifest.

exception FormUnavailable#

A dataset has no native or df form.

Subclasses AttributeError, so hasattr(dataset, "df") is False for such a dataset.

exception MissingExtraError#

A library that a loader needs is not installed.