DatasetFile

class nominal.core.DatasetFile(id, dataset_rid, name, bounds, uploaded_at, ingested_at, deleted_at, ingest_status, timestamp_channel, timestamp_type, file_tags, tag_columns)

Bases: RefreshableConjureMixin[DatasetFile]

An uploaded dataset file and its ingestion status and time bounds.

id: str

dataset_rid: str

name: str

bounds: Bounds | None

uploaded_at: IntegralNanosecondsUTC

ingested_at: IntegralNanosecondsUTC | None

deleted_at: IntegralNanosecondsUTC | None

ingest_status: IngestStatus

timestamp_channel: str | None

timestamp_type: TypedTimestampType | None

file_tags: Mapping[str, str] | None

tag_columns: Mapping[str, str] | None

delete()

Deletes the dataset file, removing its data permanently from Nominal.

Note

This cannot be undone outside of fully re-ingesting the file into Nominal.

download(output_directory, *, part_size=DEFAULT_CHUNK_SIZE, num_retries=3)

Download the dataset file to a destination on local disk.

Parameters:

• output_directory (Path | str)

  Download file to the given directory

• part_size (int)

  Size (in bytes) of chunks to use when downloading file.

• num_retries (int)

  Number of retries to perform per part download if any exception occurs

Returns:

Path that the file was downloaded to

Raises:

• FileNotFoundError – Output directory doesn’t exist

• FileExistsError – File already exists at destination

• RuntimeError – Error downloading file

Return type:

Path

download_original_files(output_directory, *, part_size=DEFAULT_CHUNK_SIZE, num_retries=3)

Download the input file(s) for a containerized extractor to a destination on local disk.

Parameters:

• output_directory (Path | str)

  Download file(s) to the given directory

• part_size (int)

  Size (in bytes) of chunks to use when downloading files.

• num_retries (int)

  Number of retries to perform per part download if any exception occurs

Returns:

Path(s) that the file(s) were downloaded to

Raises:

• NotADirectoryError – Output directory exists, but is not a directory

• FileNotFoundError – Output directory doesn’t exist

• FileExistsError – File already exists at destination

• RuntimeError – Failed to determine metadata about files to download

Return type:

Sequence[Path]

Note

Any file that fails to download will result in an error log and will not be returned as an output path

get_file_size()

Retrieves the size of the file in bytes, or None if it could not be determined

Note

This has only been extensively tested on AWS-based environments, and may fail in some self-hosted environments– a RuntimeError will be thrown in this case.

Return type:

int | None

get_ingest_error()

Returns the ingest exception if the file FAILED, or None if there is no error.

Refreshes the file from the API before checking.

Return type:

NominalIngestError | None

poll_until_ingestion_completed(interval=datetime.timedelta(seconds=1))

Block until dataset file ingestion has completed

This method polls Nominal for ingest status after uploading a file to a dataset on an interval.

Return type:

Self

refresh()

Return type:

Self