Ingest service#

The Ingest Service handles the data ingestion into Nominal/data store.

OpenAPI file: IngestService.json

Endpoints#

Ingest#

POST /ingest/v1/ingest

Triggers an ingest job, allowing either creating a new dataset or uploading to an existing one. This endpoint is meant to supersede all other ingestion endpoints as their functionality gets migrated to this one.

Body

options IngestOptions required

​

Set type to "dataflash".

dataflash DataflashOpts

​

Fields of DataflashOpts
source IngestSource required

​

Set type to "s3".

s3 S3IngestSource

​

Fields of S3IngestSource
path string required

​

Set type to "gcs".

gcs GcsIngestSource

​

Fields of GcsIngestSource
path string required

​

Set type to "presignedFile".

presignedFile PresignedFileIngestSource

​

Fields of PresignedFileIngestSource
url string required

​

Set type to "fileStore".

fileStore FileStoreIngestSource

​

Fields of FileStoreIngestSource
fileRevisionRid string (RID) required

​

target DatasetIngestTarget required

​

Set type to "new".

new NewDatasetIngestDestination

​

Fields of NewDatasetIngestDestination
datasetName string

​

datasetDescription string

​

properties map of PropertyName to string

​

labels array of string

​

channelConfig ChannelConfig

​

Fields of ChannelConfig
prefixTreeDelimiter string

If set, will construct a prefix tree for channels of the dataset using the given delimiter.

workspace string (RID)

The workspace in which to create the dataset. If not provided, the dataset will be created in the default workspace for the user’s organization, if the default workspace for the organization is configured.

markingRids array of string (RID)

The markings to apply to the created dataset. If not provided, the dataset will be visible to all users in the same workspace.

Set type to "existing".

existing ExistingDatasetIngestDestination

​

Fields of ExistingDatasetIngestDestination
datasetRid string (RID) required

​

expandRunBounds array of string (RID)

The ingest will keep track of the range of timestamps in the ingested data and update the included run’s time bounds. For runs with no end time (uninitialized), it will always set the start time to the file’s start time and set the end time to the file’s end time. For runs with existing end times, it will only expand the bounds (earliest start time, latest end time).

ignoreInvalidTimestamps boolean

If true, skips dataflash records whose timestamp cannot be represented (corrupted/out-of-range) instead of failing the entire ingest. If not provided, defaults to false.

additionalFileTags map of TagName to string

Specifies a tag set to apply to all data in the file.

Set type to "mcapProtobufTimeseries".

mcapProtobufTimeseries McapProtobufTimeseriesOpts

​

Fields of McapProtobufTimeseriesOpts
source IngestSource required

​

target DatasetIngestTarget required

​

channelFilter McapChannels required

​

Set type to "all".

all Empty

​

Set type to "include".

include array of McapChannelLocator

​

Set type to "topic".

topic string

​

Set type to "id".

id integer

​

Set type to "exclude".

exclude array of McapChannelLocator

​

timestampType McapTimestampType required

​

Set type to "logTime".

logTime LogTime

​

ignoreInvalidTopics boolean

If true, skips invalid MCAP topics during ingestion instead of failing the entire ingest. If not provided, defaults to false.

additionalFileTags map of TagName to string

Specifies a tag set to apply to all data in the file.

Set type to "journalJson".

journalJson JournalJsonOpts

​

Fields of JournalJsonOpts
source IngestSource required

​

target DatasetIngestTarget required

​

channel string

If provided, ingests logs to the given channel. By default, log data will be ingested to a channel named ‘logs’.

timestampMetadata JournalTimestampMetadata

Optional timestamp field for the JSON lines. When provided, each line is timestamped from this field instead of the journald default __REALTIME_TIMESTAMP, allowing non-journald JSON logs to ingest. Each line must still contain a MESSAGE field. Only numeric epoch timestamps are supported. When absent, the journald default (__REALTIME_TIMESTAMP, microseconds) is used.

Fields of JournalTimestampMetadata
fieldName string required

​

epochOfTimeUnit EpochTimestamp required

​

Fields of EpochTimestamp
timeUnit TimeUnit required

​

Values of TimeUnit
DAYS

​

HOURS

​

MINUTES

​

SECONDS

​

MILLISECONDS

​

MICROSECONDS

​

NANOSECONDS

​

Set type to "csv".

csv CsvOpts

​

Fields of CsvOpts
source IngestSource required

​

target DatasetIngestTarget required

​

timestampMetadata TimestampMetadata required

​

Fields of TimestampMetadata
seriesName string required

​

timestampType TimestampType required

​

Set type to "relative".

relative RelativeTimestamp

​

Fields of RelativeTimestamp
timeUnit TimeUnit required

​

offset string (date-time)

Starting timestamp to use when indexing the file. This field is required when uploading a new file to an existing dataset.

Set type to "absolute".

absolute AbsoluteTimestamp

​

Set type to "iso8601".

iso8601 Iso8601Timestamp

​

Set type to "epochOfTimeUnit".

epochOfTimeUnit EpochTimestamp

​

Set type to "customFormat".

customFormat CustomTimestamp

​

Fields of CustomTimestamp
format string required

The format string should be in the format of the DateTimeFormatter class in Java.

defaultYear integer

Default year is accepted as an optional field for cases like IRIG time format and will be overridden by year in time format.

defaultDayOfYear integer

Default day of year is accepted as an optional field for cases like IRIG time format and will be overridden by day of year in time format.

channelPrefix string required

​

channelNameOverrides map of ColumnName to string

​

tagKeysFromColumns array of string deprecated

Deprecated: Deprecated in favor of tagColumns.

tagColumns map of TagName to string

A map of tag names to column names to derive the tag values from.

additionalFileTags map of TagName to string

Specifies a tag set to apply to all data in the file.

excludeColumns array of string

A set of column names to exclude from ingestion. These columns will not be ingested as channels. Useful for excluding columns that contain unsupported data types like multidimensional arrays.

Set type to "parquet".

parquet ParquetOpts

​

Fields of ParquetOpts
source IngestSource required

​

target DatasetIngestTarget required

​

timestampMetadata TimestampMetadata required

​

channelPrefix string required

​

channelNameOverrides map of ColumnName to string

​

tagKeysFromColumns array of string deprecated

Deprecated: Deprecated in favor of tagColumns.

tagColumns map of TagName to string

A map of tag names to column names to derive the tag values from.

additionalFileTags map of TagName to string

Specifies a tag set to apply to all data in the file.

isArchive boolean

If true, the file is an archive. Supported archive formats include .tar, .tar.gz, and .zip. Only files ending in .parquet within the archive will be ingested. If field not provided, defaults to false.

excludeColumns array of string

A set of column names to exclude from ingestion. These columns will not be ingested as channels. Useful for excluding columns that contain unsupported data types like multidimensional arrays.

Set type to "video".

video VideoOpts deprecated

Deprecated: Legacy Video-entity ingest. Use videoV2, which ingests video as a channel on a dataset.

Fields of VideoOpts
source IngestSource required

​

target VideoIngestTarget required

​

Set type to "new".

new NewVideoIngestDestination

​

Fields of NewVideoIngestDestination
title string

Title of the Video that will get created. If not provided, a name is deduced from the ingested file.

description string

Description that is applied to the newly created video

properties map of PropertyName to string

Key-Value properties that are applied to the newly created video

labels array of string

Labels that are applied to the newly created video

videoFileDetails VideoFileIngestDetails

Metadata to associate with any created video file

Fields of VideoFileIngestDetails
fileDescription string

Description that is applied to the newly created video file. If not provided, defaults to the description used to create the video.

fileProperties map of PropertyName to string deprecated

Key-Value properties that are applied to the newly created video file. Deprecated: Field is ignored as video files do not have properties. Will be removed after 4/5/2025

fileLabels array of string deprecated

Labels that are applied to the newly created video file. Deprecated: Field is ignored as video files do not have labels. Will be removed after 4/5/2025

workspace string (RID)

The workspace in which to create the video. If not provided, the video will be created in the default workspace for the user’s organization, if the default workspace for the organization is configured.

markingRids array of string (RID)

The markings to apply to the created video. If not provided, the video will be visible to all users in the same workspace.

Set type to "existing".

existing ExistingVideoIngestDestination

​

Fields of ExistingVideoIngestDestination
videoRid string (RID) required

RID of the video to ingest the newly created video file to.

videoFileDetails VideoFileIngestDetails

Metadata to associate with any created video file

timestampManifest VideoFileTimestampManifest required

​

Set type to "mcap".

mcap McapTimestampManifest

​

Fields of McapTimestampManifest
mcapChannelLocator McapChannelLocator required

​

Set type to "s3path".

s3path string

​

Set type to "s3paths".

s3paths array of string

Multiple sidecar manifest files whose per-frame timestamps are concatenated in list order. Used when a video’s frame timestamps are recorded across several files.

Set type to "noManifest".

noManifest video.api.NoTimestampManifest

​

Fields of video.api.NoTimestampManifest
startingTimestamp Timestamp required

Specifies the original starting timestamp of the video.

Fields of Timestamp
seconds integer (int64) required

​

nanos integer (int64) required

​

scaleParameter video.api.ScaleParameter

A field that specifies that the frame rate of the video does not match the frame rate of the camera | i.e. a slowed down or sped up video. Can specify either the camera frame rate or the absolute end time.

Set type to "trueFrameRate".

trueFrameRate number

​

Set type to "endingTimestamp".

endingTimestamp Timestamp

​

Set type to "scaleFactor".

scaleFactor number

the scale factor can be used to calculate whether media duration differs from a video’s | real duration, and if so, the true frame rate of the camera. The video time will thus be scaled | by the ratio of the real duration to media duration, or media frame rate to true frame rate.

overWriteSegments boolean

If true, overlapping segments from other video files within the same video will be deleted before inserting new segments. The cached segment metadata for affected files will be recomputed.

Set type to "videoV2".

videoV2 VideoOptsV2

​

Fields of VideoOptsV2
source IngestSource required

​

target DatasetIngestTarget required

​

timestampManifest VideoFileTimestampManifest required

​

channel string required

​

tags map of TagName to string

​

overWriteSegments boolean

If true, overlapping segments from other dataset files within the same series will be deleted before inserting new segments. The cached segment metadata for affected files will be recomputed.

Set type to "containerized".

containerized ContainerizedOpts

​

Fields of ContainerizedOpts
sources map of EnvironmentVariable to IngestSource

​

arguments map of EnvironmentVariable to string

​

extractorRid string (RID) required

​

timestampMetadata TimestampMetadata

​

tag string

​

target DatasetIngestTarget required

​

additionalFileTags map of TagName to string

Specifies a tag set to apply to all data in the file.

Set type to "avroStream".

avroStream AvroStreamOpts

​

Fields of AvroStreamOpts
source IngestSource required

​

target DatasetIngestTarget required

​

additionalFileTags map of TagName to string

Specifies a tag set to apply to all data in the file.

timestampType AvroNumericTimestampType

How to interpret the numeric values in the timestamps array. Defaults to epoch-nanoseconds when omitted, matching the original contract of this API.

Set type to "epoch".

epoch EpochTimestamp

​

Set type to "relative".

relative RelativeTimestamp

​

Set type to "pointCloud".

pointCloud PointCloudOpts

​

Fields of PointCloudOpts
source IngestSource required

​

target PointCloudIngestTarget required

​

Set type to "existing".

existing ExistingSpatialIngestDestination

​

Fields of ExistingSpatialIngestDestination
spatialRid string (RID) required

​

daggerImportConfig any JSON value required

The Dagger **v2** ImportRequest body — everything from that schema except source_uri, which Nominal fills in from the resolved presigned URL for source. Forwarded to Dagger’s POST /v2/imports/{model_uuid}. Construct via the language’s generated dagger-client types (e.g. dagger_client.models.ImportRequest in Python). Carries the required archetype (attribute definitions and sampler configuration), an optional format (the source format and its format-specific parameters, including the geometry_type and columns a CSV source needs; omit it to let Dagger infer the format from the file extension), and an optional quiche_config — all of which are domain knowledge the client owns when it produced the file. The legacy v1 shape, with geometry_type and columns at the top level, is rejected.

sensorMetadata PointCloudMetadata

Sensor metadata to associate with the point cloud.

Fields of PointCloudMetadata
sensorModel string

The sensor model name, e.g. “Velodyne VLP-16”, “Ouster OS1-128”.

coordinateSystem string

Coordinate reference system identifier, e.g. “EPSG:4326”, “ENU”.

resolutionMm number

Spatial resolution of the point cloud in millimeters.

scanPattern ScanPattern

The scan pattern of the lidar sensor.

Values of ScanPattern
ROTATING

​

SOLID_STATE

​

FLASH

​

MECHANICAL

​

channel string required

Channel name for workbook integration.

tags map of TagName to string

Tags to apply to the point cloud series.

Response

details IngestDetails

​

Set type to "dataset".

dataset IngestDatasetFileDetails

​

Fields of IngestDatasetFileDetails
datasetFileId string (UUID)

​

datasetRid string (RID)

​

Set type to "video".

video IngestVideoFileDetails deprecated

Deprecated: Returned only for legacy Video-entity ingests; videoV2 ingests return dataset details carrying the created dataset file id.

Fields of IngestVideoFileDetails
videoFileRid string (RID)

​

videoRid string (RID)

​

Set type to "spatial".

spatial IngestSpatialDetails

​

Fields of IngestSpatialDetails
spatialRid string (RID)

​

ingestJobRid string (RID)

​

Request

curl -X POST "https://api.gov.nominal.io/api/ingest/v1/ingest" \
  -H "Authorization: Bearer <api key>" \
  -H "Content-Type: application/json" \
  --data '{
  "options": {
    "type": "dataflash",
    "dataflash": {
      "source": {
        "type": "s3",
        "s3": {
          "path": "<path>"
        }
      },
      "target": {
        "type": "new",
        "new": {
          "datasetName": "<dataset name>",
          "datasetDescription": "<dataset description>",
          "properties": {
            "<key>": "<property value>"
          },
          "labels": [
            "<label>"
          ],
          "channelConfig": {
            "prefixTreeDelimiter": "<prefix tree delimiter>"
          },
          "workspace": "<workspace rid>",
          "markingRids": [
            "<marking rid>"
          ]
        }
      }
    }
  }
}'

import requests

response = requests.post(
    "https://api.gov.nominal.io/api/ingest/v1/ingest",
    headers={"Authorization": "Bearer <api key>"},
    json={
        "options": {
            "type": "dataflash",
            "dataflash": {
                "source": {"type": "s3", "s3": {"path": "<path>"}},
                "target": {
                    "type": "new",
                    "new": {
                        "datasetName": "<dataset name>",
                        "datasetDescription": "<dataset description>",
                        "properties": {"<key>": "<property value>"},
                        "labels": ["<label>"],
                        "channelConfig": {
                            "prefixTreeDelimiter": "<prefix tree delimiter>",
                        },
                        "workspace": "<workspace rid>",
                        "markingRids": ["<marking rid>"],
                    },
                },
            },
        },
    },
)
response.raise_for_status()
print(response.json())

import httpx

response = httpx.post(
    "https://api.gov.nominal.io/api/ingest/v1/ingest",
    headers={"Authorization": "Bearer <api key>"},
    json={
        "options": {
            "type": "dataflash",
            "dataflash": {
                "source": {"type": "s3", "s3": {"path": "<path>"}},
                "target": {
                    "type": "new",
                    "new": {
                        "datasetName": "<dataset name>",
                        "datasetDescription": "<dataset description>",
                        "properties": {"<key>": "<property value>"},
                        "labels": ["<label>"],
                        "channelConfig": {
                            "prefixTreeDelimiter": "<prefix tree delimiter>",
                        },
                        "workspace": "<workspace rid>",
                        "markingRids": ["<marking rid>"],
                    },
                },
            },
        },
    },
)
response.raise_for_status()
print(response.json())

import asyncio

import aiohttp


async def main() -> None:
    async with aiohttp.ClientSession() as session:
        async with session.post(
            "https://api.gov.nominal.io/api/ingest/v1/ingest",
            headers={"Authorization": "Bearer <api key>"},
            json={
                "options": {
                    "type": "dataflash",
                    "dataflash": {
                        "source": {"type": "s3", "s3": {"path": "<path>"}},
                        "target": {
                            "type": "new",
                            "new": {
                                "datasetName": "<dataset name>",
                                "datasetDescription": "<dataset description>",
                                "properties": {"<key>": "<property value>"},
                                "labels": ["<label>"],
                                "channelConfig": {
                                    "prefixTreeDelimiter": "<prefix tree delimiter>",
                                },
                                "workspace": "<workspace rid>",
                                "markingRids": ["<marking rid>"],
                            },
                        },
                    },
                },
            },
        ) as response:
            response.raise_for_status()
            print(await response.json())


asyncio.run(main())

Response: 200

{
  "details": {
    "type": "dataset",
    "dataset": {
      "datasetFileId": "<dataset file id>",
      "datasetRid": "<dataset rid>"
    }
  },
  "ingestJobRid": "<ingest job rid>"
}

Rerun ingest#

POST /ingest/v1/re-ingest

Triggers an ingest job using an existing ingest job RID. Returns the same response format as the /ingest endpoint.

Only a job that has finished can be re-run: re-running one still in flight races or duplicates the run already in progress. Cancel it first.

Body

ingestJobRid string (RID) required

​

Response

details IngestDetails

​

Set type to "dataset".

dataset IngestDatasetFileDetails

​

Fields of IngestDatasetFileDetails
datasetFileId string (UUID)

​

datasetRid string (RID)

​

Set type to "video".

video IngestVideoFileDetails deprecated

Deprecated: Returned only for legacy Video-entity ingests; videoV2 ingests return dataset details carrying the created dataset file id.

Fields of IngestVideoFileDetails
videoFileRid string (RID)

​

videoRid string (RID)

​

Set type to "spatial".

spatial IngestSpatialDetails

​

Fields of IngestSpatialDetails
spatialRid string (RID)

​

ingestJobRid string (RID)

​

Request

curl -X POST "https://api.gov.nominal.io/api/ingest/v1/re-ingest" \
  -H "Authorization: Bearer <api key>" \
  -H "Content-Type: application/json" \
  --data '{
  "ingestJobRid": "<ingest job rid>"
}'

import requests

response = requests.post(
    "https://api.gov.nominal.io/api/ingest/v1/re-ingest",
    headers={"Authorization": "Bearer <api key>"},
    json={"ingestJobRid": "<ingest job rid>"},
)
response.raise_for_status()
print(response.json())

import httpx

response = httpx.post(
    "https://api.gov.nominal.io/api/ingest/v1/re-ingest",
    headers={"Authorization": "Bearer <api key>"},
    json={"ingestJobRid": "<ingest job rid>"},
)
response.raise_for_status()
print(response.json())

import asyncio

import aiohttp


async def main() -> None:
    async with aiohttp.ClientSession() as session:
        async with session.post(
            "https://api.gov.nominal.io/api/ingest/v1/re-ingest",
            headers={"Authorization": "Bearer <api key>"},
            json={"ingestJobRid": "<ingest job rid>"},
        ) as response:
            response.raise_for_status()
            print(await response.json())


asyncio.run(main())

Response: 200

{
  "details": {
    "type": "dataset",
    "dataset": {
      "datasetFileId": "<dataset file id>",
      "datasetRid": "<dataset rid>"
    }
  },
  "ingestJobRid": "<ingest job rid>"
}

Ingest run#

POST /ingest/v1/ingest-run

Creates a run and ingests data sources to be added to the run.

Body

rid string (RID)

If a run with the same rid already exists, the run will be updated.

title string required

​

description string required

​

startTime UtcTimestamp required

​

Fields of UtcTimestamp
secondsSinceEpoch integer (int64) required

​

offsetNanoseconds integer (int64)

​

endTime UtcTimestamp

​

properties map of PropertyName to string

​

labels array of string

​

runPrefix string

for example, SIM, HTL, FLT

dataSources map of DataSourceRefName to IngestRunDataSource

​

Fields of IngestRunDataSource
dataSource IngestDataSource required

​

Set type to "existingDataSource".

existingDataSource string (RID)

​

Set type to "newDataSource".

newDataSource DeprecatedNewDataSource

​

Set type to "csv".

csv DeprecatedNewCsv

​

Fields of DeprecatedNewCsv
source IngestSource required

​

Set type to "s3".

s3 S3IngestSource

​

Fields of S3IngestSource
path string required

​

Set type to "gcs".

gcs GcsIngestSource

​

Fields of GcsIngestSource
path string required

​

Set type to "presignedFile".

presignedFile PresignedFileIngestSource

​

Fields of PresignedFileIngestSource
url string required

​

Set type to "fileStore".

fileStore FileStoreIngestSource

​

Fields of FileStoreIngestSource
fileRevisionRid string (RID) required

​

name string

​

properties map of string to string

​

timeColumnSpec TimestampMetadata

​

Fields of TimestampMetadata
seriesName string required

​

timestampType TimestampType required

​

Set type to "relative".

relative RelativeTimestamp

​

Fields of RelativeTimestamp
timeUnit TimeUnit required

​

Values of TimeUnit
DAYS

​

HOURS

​

MINUTES

​

SECONDS

​

MILLISECONDS

​

MICROSECONDS

​

NANOSECONDS

​

offset string (date-time)

Starting timestamp to use when indexing the file. This field is required when uploading a new file to an existing dataset.

Set type to "absolute".

absolute AbsoluteTimestamp

​

Set type to "iso8601".

iso8601 Iso8601Timestamp

​

Set type to "epochOfTimeUnit".

epochOfTimeUnit EpochTimestamp

​

Fields of EpochTimestamp
timeUnit TimeUnit required

​

Set type to "customFormat".

customFormat CustomTimestamp

​

Fields of CustomTimestamp
format string required

The format string should be in the format of the DateTimeFormatter class in Java.

defaultYear integer

Default year is accepted as an optional field for cases like IRIG time format and will be overridden by year in time format.

defaultDayOfYear integer

Default day of year is accepted as an optional field for cases like IRIG time format and will be overridden by day of year in time format.

channelConfig ChannelConfig

​

Fields of ChannelConfig
prefixTreeDelimiter string

If set, will construct a prefix tree for channels of the dataset using the given delimiter.

Set type to "newDataSourceV2".

newDataSourceV2 NewDataSource

​

Fields of NewDataSource
source IngestSource required

​

properties map of PropertyName to string

​

labels array of string

​

description string

​

name string

​

timeColumnSpec TimestampMetadata

​

channelConfig ChannelConfig

​

timeOffsetSpec TimeOffsetSpec

​

Set type to "nanos".

nanos Duration

​

Fields of Duration
seconds integer (int64) required

​

nanos integer (int64) required

​

workspace string (RID)

The workspace in which to create the dataset. If not provided, the dataset will be created in the default workspace for the user’s organization, if the default workspace for the organization is configured.

Response

runRid string (RID)

​

Request

curl -X POST "https://api.gov.nominal.io/api/ingest/v1/ingest-run" \
  -H "Authorization: Bearer <api key>" \
  -H "Content-Type: application/json" \
  --data '{
  "title": "<title>",
  "description": "<description>",
  "startTime": {
    "secondsSinceEpoch": 1
  }
}'

import requests

response = requests.post(
    "https://api.gov.nominal.io/api/ingest/v1/ingest-run",
    headers={"Authorization": "Bearer <api key>"},
    json={
        "title": "<title>",
        "description": "<description>",
        "startTime": {"secondsSinceEpoch": 1},
    },
)
response.raise_for_status()
print(response.json())

import httpx

response = httpx.post(
    "https://api.gov.nominal.io/api/ingest/v1/ingest-run",
    headers={"Authorization": "Bearer <api key>"},
    json={
        "title": "<title>",
        "description": "<description>",
        "startTime": {"secondsSinceEpoch": 1},
    },
)
response.raise_for_status()
print(response.json())

import asyncio

import aiohttp


async def main() -> None:
    async with aiohttp.ClientSession() as session:
        async with session.post(
            "https://api.gov.nominal.io/api/ingest/v1/ingest-run",
            headers={"Authorization": "Bearer <api key>"},
            json={
                "title": "<title>",
                "description": "<description>",
                "startTime": {"secondsSinceEpoch": 1},
            },
        ) as response:
            response.raise_for_status()
            print(await response.json())


asyncio.run(main())

Response: 200

{
  "runRid": "<run rid>"
}

Ingest video#

POST /ingest/v1/ingest-video

Deprecated: Replaced by ingest.

Ingests video data from a S3 Nominal upload bucket.

Body

sources array of IngestSource

​

Set type to "s3".

s3 S3IngestSource

​

Fields of S3IngestSource
path string required

​

Set type to "gcs".

gcs GcsIngestSource

​

Fields of GcsIngestSource
path string required

​

Set type to "presignedFile".

presignedFile PresignedFileIngestSource

​

Fields of PresignedFileIngestSource
url string required

​

Set type to "fileStore".

fileStore FileStoreIngestSource

​

Fields of FileStoreIngestSource
fileRevisionRid string (RID) required

​

properties map of PropertyName to string

​

labels array of string

​

title string

​

description string

​

timestamps VideoTimestampManifest required

​

Set type to "noManifest".

noManifest ingest.api.NoTimestampManifest

​

Fields of ingest.api.NoTimestampManifest
startingTimestamp UtcTimestamp required

​

Fields of UtcTimestamp
secondsSinceEpoch integer (int64) required

​

offsetNanoseconds integer (int64)

​

scaleParameter ingest.api.ScaleParameter

A field that specifies that the frame rate of the video does not match the frame rate of the camera | i.e. a slowed down or sped up video. Can specify either the camera frame rate or the absolute end time.

Set type to "trueFrameRate".

trueFrameRate number

​

Set type to "endingTimestamp".

endingTimestamp UtcTimestamp

the timestamp corresponding to absolute starting timestamp plus absolute duration of the video.

Set type to "scaleFactor".

scaleFactor number

the scale factor can be used to calculate whether media duration differs from a video’s | real duration, and if so, the true frame rate of the camera. The video time will thus be scaled | by the ratio of the real duration to media duration, or media frame rate to true frame rate.

Set type to "timestampManifests".

timestampManifests TimestampManifest

​

Fields of TimestampManifest
sources array of IngestSource

​

workspace string (RID)

The workspace in which to create the video. If not provided, the video will be created in the default workspace for the user’s organization, if the default workspace for the organization is configured.

Response

videoRid string (RID)

​

videoFileRid string (RID)

​

asyncHandle AsyncHandle deprecated

Deprecated: Deprecated

Fields of AsyncHandle
rid string (RID)

​

Request

curl -X POST "https://api.gov.nominal.io/api/ingest/v1/ingest-video" \
  -H "Authorization: Bearer <api key>" \
  -H "Content-Type: application/json" \
  --data '{
  "timestamps": {
    "type": "noManifest",
    "noManifest": {
      "startingTimestamp": {
        "secondsSinceEpoch": 1
      }
    }
  }
}'

import requests

response = requests.post(
    "https://api.gov.nominal.io/api/ingest/v1/ingest-video",
    headers={"Authorization": "Bearer <api key>"},
    json={
        "timestamps": {
            "type": "noManifest",
            "noManifest": {"startingTimestamp": {"secondsSinceEpoch": 1}},
        },
    },
)
response.raise_for_status()
print(response.json())

import httpx

response = httpx.post(
    "https://api.gov.nominal.io/api/ingest/v1/ingest-video",
    headers={"Authorization": "Bearer <api key>"},
    json={
        "timestamps": {
            "type": "noManifest",
            "noManifest": {"startingTimestamp": {"secondsSinceEpoch": 1}},
        },
    },
)
response.raise_for_status()
print(response.json())

import asyncio

import aiohttp


async def main() -> None:
    async with aiohttp.ClientSession() as session:
        async with session.post(
            "https://api.gov.nominal.io/api/ingest/v1/ingest-video",
            headers={"Authorization": "Bearer <api key>"},
            json={
                "timestamps": {
                    "type": "noManifest",
                    "noManifest": {"startingTimestamp": {"secondsSinceEpoch": 1}},
                },
            },
        ) as response:
            response.raise_for_status()
            print(await response.json())


asyncio.run(main())

Response: 200

{
  "videoRid": "<video rid>",
  "videoFileRid": "<video file rid>",
  "asyncHandle": {
    "rid": "<rid>"
  }
}

Reingest from datasets#

POST /ingest/v1/reingest-dataset-files

Re-ingests data from provided source datasets into either an existing target dataset, or a new one. Only supported for CSV and Parquet dataset files. Will only reingest dataset files and will drop streaming data from datasets.

Body

sourceDatasets array of string (RID)

The datasets to reingest data from. Can only include tagless, non-streaming datasets (V1). The datasets must be of the same granularity and must only include CSV or Parquet files. Will attempt to reingest from datasets in list order.

targetDataset DatasetIngestTarget required

The dataset to ingest data into. Can either be a new dataset or an existing dataset RID.

Set type to "new".

new NewDatasetIngestDestination

​

Fields of NewDatasetIngestDestination
datasetName string

​

datasetDescription string

​

properties map of PropertyName to string

​

labels array of string

​

channelConfig ChannelConfig

​

Fields of ChannelConfig
prefixTreeDelimiter string

If set, will construct a prefix tree for channels of the dataset using the given delimiter.

workspace string (RID)

The workspace in which to create the dataset. If not provided, the dataset will be created in the default workspace for the user’s organization, if the default workspace for the organization is configured.

markingRids array of string (RID)

The markings to apply to the created dataset. If not provided, the dataset will be visible to all users in the same workspace.

Set type to "existing".

existing ExistingDatasetIngestDestination

​

Fields of ExistingDatasetIngestDestination
datasetRid string (RID) required

​

expandRunBounds array of string (RID)

The ingest will keep track of the range of timestamps in the ingested data and update the included run’s time bounds. For runs with no end time (uninitialized), it will always set the start time to the file’s start time and set the end time to the file’s end time. For runs with existing end times, it will only expand the bounds (earliest start time, latest end time).

ingestMetadata map of DatasetRid to IngestMetadata

Mapping of dataset to ingest metadata. Ingest metadata only needs to be provided in request if it cannot be recovered from prior ingests. See documentation on each field to determine what metadata can be auto-recovered.

Fields of IngestMetadata
timestampMetadata TimestampMetadata

The timestamp metadata will be recovered from the dataset files if possible. Older datasets may have unrecoverable timestamp metadata. If unrecoverable, reingestion will throw MissingMetadataForReingest if not provided in request.

Fields of TimestampMetadata
seriesName string required

​

timestampType TimestampType required

​

Set type to "relative".

relative RelativeTimestamp

​

Fields of RelativeTimestamp
timeUnit TimeUnit required

​

Values of TimeUnit
DAYS

​

HOURS

​

MINUTES

​

SECONDS

​

MILLISECONDS

​

MICROSECONDS

​

NANOSECONDS

​

offset string (date-time)

Starting timestamp to use when indexing the file. This field is required when uploading a new file to an existing dataset.

Set type to "absolute".

absolute AbsoluteTimestamp

​

Set type to "iso8601".

iso8601 Iso8601Timestamp

​

Set type to "epochOfTimeUnit".

epochOfTimeUnit EpochTimestamp

​

Fields of EpochTimestamp
timeUnit TimeUnit required

​

Set type to "customFormat".

customFormat CustomTimestamp

​

Fields of CustomTimestamp
format string required

The format string should be in the format of the DateTimeFormatter class in Java.

defaultYear integer

Default year is accepted as an optional field for cases like IRIG time format and will be overridden by year in time format.

defaultDayOfYear integer

Default day of year is accepted as an optional field for cases like IRIG time format and will be overridden by day of year in time format.

channelPrefix string

Channel prefix to use when reingesting the dataset. Defaults to empty string. Not recoverable from prior ingests and must be provided in request if needed.

channelNameOverrides map of ColumnName to string

Renames individual columns to explicit channel names. Composes with channelPrefix: final name = channelPrefix + channelNameOverrides.getOrDefault(column, column). Defaults to empty map. Not recoverable from prior ingests and must be provided in request if needed. At most 1000 entries may be provided. Resolved channel names must be unique across columns; renaming a column onto another column’s name fails the ingest.

tagColumns map of TagName to string

A map of tag names to column names to derive the tag values from. Not recoverable from prior ingests and must be provided in request if needed.

additionalFileTags map of TagName to string

Additional tags to apply to all dataset files within the the given dataset. Not recoverable from prior ingests and must be provided in request if needed.

Response

datasets map of DatasetRid to ReingestFilesMapping

​

Fields of ReingestFilesMapping
fileMap map of DatasetFileId to IngestDetails

​

Set type to "dataset".

dataset IngestDatasetFileDetails

​

Fields of IngestDatasetFileDetails
datasetFileId string (UUID)

​

datasetRid string (RID)

​

Set type to "video".

video IngestVideoFileDetails deprecated

Deprecated: Returned only for legacy Video-entity ingests; videoV2 ingests return dataset details carrying the created dataset file id.

Fields of IngestVideoFileDetails
videoFileRid string (RID)

​

videoRid string (RID)

​

Set type to "spatial".

spatial IngestSpatialDetails

​

Fields of IngestSpatialDetails
spatialRid string (RID)

​

Request

curl -X POST "https://api.gov.nominal.io/api/ingest/v1/reingest-dataset-files" \
  -H "Authorization: Bearer <api key>" \
  -H "Content-Type: application/json" \
  --data '{
  "targetDataset": {
    "type": "new",
    "new": {
      "datasetName": "<dataset name>",
      "datasetDescription": "<dataset description>",
      "properties": {
        "<key>": "<property value>"
      },
      "labels": [
        "<label>"
      ],
      "channelConfig": {
        "prefixTreeDelimiter": "<prefix tree delimiter>"
      },
      "workspace": "<workspace rid>",
      "markingRids": [
        "<marking rid>"
      ]
    }
  }
}'

import requests

response = requests.post(
    "https://api.gov.nominal.io/api/ingest/v1/reingest-dataset-files",
    headers={"Authorization": "Bearer <api key>"},
    json={
        "targetDataset": {
            "type": "new",
            "new": {
                "datasetName": "<dataset name>",
                "datasetDescription": "<dataset description>",
                "properties": {"<key>": "<property value>"},
                "labels": ["<label>"],
                "channelConfig": {"prefixTreeDelimiter": "<prefix tree delimiter>"},
                "workspace": "<workspace rid>",
                "markingRids": ["<marking rid>"],
            },
        },
    },
)
response.raise_for_status()
print(response.json())

import httpx

response = httpx.post(
    "https://api.gov.nominal.io/api/ingest/v1/reingest-dataset-files",
    headers={"Authorization": "Bearer <api key>"},
    json={
        "targetDataset": {
            "type": "new",
            "new": {
                "datasetName": "<dataset name>",
                "datasetDescription": "<dataset description>",
                "properties": {"<key>": "<property value>"},
                "labels": ["<label>"],
                "channelConfig": {"prefixTreeDelimiter": "<prefix tree delimiter>"},
                "workspace": "<workspace rid>",
                "markingRids": ["<marking rid>"],
            },
        },
    },
)
response.raise_for_status()
print(response.json())

import asyncio

import aiohttp


async def main() -> None:
    async with aiohttp.ClientSession() as session:
        async with session.post(
            "https://api.gov.nominal.io/api/ingest/v1/reingest-dataset-files",
            headers={"Authorization": "Bearer <api key>"},
            json={
                "targetDataset": {
                    "type": "new",
                    "new": {
                        "datasetName": "<dataset name>",
                        "datasetDescription": "<dataset description>",
                        "properties": {"<key>": "<property value>"},
                        "labels": ["<label>"],
                        "channelConfig": {
                            "prefixTreeDelimiter": "<prefix tree delimiter>",
                        },
                        "workspace": "<workspace rid>",
                        "markingRids": ["<marking rid>"],
                    },
                },
            },
        ) as response:
            response.raise_for_status()
            print(await response.json())


asyncio.run(main())

Response: 200

{
  "datasets": {
    "<key>": {
      "fileMap": {
        "<key>": {
          "type": "dataset",
          "dataset": {
            "datasetFileId": "<dataset file id>",
            "datasetRid": "<dataset rid>"
          }
        }
      }
    }
  }
}

Delete file#

DELETE /ingest/v1/delete-file/{datasetRid}/file/{fileId}

This is a best effort deletion of the file’s data based on the ingestedAt timestamp. This is an unreversible action. Only v2 dataset file deletion is supported. !!!WARNING!!! It’s possible that the file has overwritten points, in which case, those older points will not be recovered. Only use this endpoint if this is acceptable, the data across files are disjoint, or you’re willing to re-ingest files to manually recover older points.

Path parameters

datasetRid string (RID) required

​

fileId string (UUID) required

​

Response

No content (HTTP 204).

Request

curl -X DELETE "https://api.gov.nominal.io/api/ingest/v1/delete-file/<dataset rid>/file/<file id>" \
  -H "Authorization: Bearer <api key>"

import requests

response = requests.delete(
    "https://api.gov.nominal.io/api/ingest/v1/delete-file/<dataset rid>/file/<file id>",
    headers={"Authorization": "Bearer <api key>"},
)
response.raise_for_status()

import httpx

response = httpx.delete(
    "https://api.gov.nominal.io/api/ingest/v1/delete-file/<dataset rid>/file/<file id>",
    headers={"Authorization": "Bearer <api key>"},
)
response.raise_for_status()

import asyncio

import aiohttp


async def main() -> None:
    async with aiohttp.ClientSession() as session:
        async with session.delete(
            "https://api.gov.nominal.io/api/ingest/v1/delete-file/<dataset rid>/file/<file id>",
            headers={"Authorization": "Bearer <api key>"},
        ) as response:
            response.raise_for_status()


asyncio.run(main())

Response: 204

No content.

Errors#

Errors shows how these errors are sent.

Each error’s errorName is UploadService: followed by its name below.

Status

Name

Notes

400

EmptyMultipartUpload

400

IncorrectSizeBytes

400

NullKey

400

SignDownloadPathOutsideUploadsBucket

400

UploadBucketNotAllowed

A multipart-upload follow-up call specified a bucket this service does not recognize. It must be the value returned as bucket by initiateMultipartUpload, unchanged.

400

WorkspaceResourceMismatch

Both workspace and resource were set on the upload request, but the workspace derived from the resource does not match the explicitly-supplied workspace. Returned as a defensive sanity check so callers don’t accidentally believe they’re uploading to one workspace while the file is actually being routed to another.

403

SignDownloadUnauthorized

404

ResourceNotFoundOrUnauthorized

The resource referenced by an upload’s resource field was not found or the caller is not authorized to access it. Returned as NOT_FOUND (rather than PERMISSION_DENIED) to avoid leaking existence information about resources the caller cannot see.

404

UploadObjectNotFound

A multipart-upload follow-up call named a File Store object the ledger has no row for. A malformed key and an unknown one are reported identically, so a caller cannot tell which of the two it supplied.

Each error’s errorName is IngestService: followed by its name below.

Status

Name

Notes

400

BatchGranularityMismatch

All dataset granularities must match.

400

ContainerizedExtractorTimestampMetadataMissing

Neither the containerized extractor (or its active container image) nor the ingest request provides timestamp metadata, so the extractor output cannot be indexed. Register the image with default timestamp metadata or supply it on the request.

400

FailedToDownloadFileUserError

400

FileDeletionNotSupported

The dataset file is part of a multi-file ingest job and cannot be deleted on its own. Delete the entire ingest job instead.

400

FrameCountTimestampsMismatch

400

IncompatibleConnectionTypeAndLogQuery

400

IncompatibleFileForDataset

400

IncompatibleInputs

The inputs provided to the ingest job are incompatible with each other

400

IngestJobNotCompatibleForRerun

The ingest job exists but is not compatible to be re-run.

400

InvalidBucket

400

InvalidCustomTimestampFormat

400

InvalidS3Path

400

InvalidUrl

400

MissingMetadataForReingest

The timestamp metadata required to reingest the dataset is missing and must be provided in the request.

400

ParallelReingestIntoDatasetV1

Cannot ingest multiple datasets in parallel into older datasets. It is recommended to reingest all source datasets and the target dataset into a new dataset destination.

400

RecursiveReingest

The target dataset is included in the source datasets.

400

ReingestTooManyFiles

The number of files to reingest is too large.

400

RunBoundsInverted

400

UnsupportedIngestSource

400

UnsupportedRequestTypeForReIngest

403

ContainerizedExtractorUnauthorizedPasswordAccess

The user is not authorized to access the containerized extractor password.

404

DatasetFileNotFound

The target dataset file could not be found.

404

DatasetNotFound

The target dataset could not be found.

404

IngestJobNotFound

404

IngestTransformNotFound

The ingest transform does not exist, or its parent ingest job is not visible to the caller. Returned as NOT_FOUND (rather than PERMISSION_DENIED) to avoid leaking existence information about transforms the caller cannot see.

404

S3PathNotFound

404

StreamingSessionNotFound

The streaming session could not be found.

409

IngestJobNotCancellable

The ingest job exists but is in a terminal state (COMPLETED, FAILED, or CANCELLED) and therefore cannot be cancelled.

500

ContainerizedExtractorContainerFailed

500

FailedToDownloadFile

500

IngestJobMissingDatasetForIdempotencyKey

A prior ingest job already exists for this idempotency key but never produced a dataset (the original submit did not get far enough). The idempotency key deduplicates to that existing job and does not re-run ingestion; rerun the job identified by ingestJobRid to complete it.

Types#

AbsoluteTimestamp#

Set type to the name of one of the other fields, and set only that field.

iso8601 Iso8601Timestamp

​

epochOfTimeUnit EpochTimestamp

​

customFormat CustomTimestamp

​

AsyncHandle#

rid string (RID) required

​

AvroNumericTimestampType#

Describes how the numeric timestamps column in an Avro Stream file should be interpreted. The Avro schema models timestamps as an array of longs, so only numeric timestamp variants are supported (no ISO-8601 / custom string formats).

Set type to the name of one of the other fields, and set only that field.

epoch EpochTimestamp

​

relative RelativeTimestamp

​

AvroStreamOpts#

Options for ingesting Avro data with the following schema. This is a “stream-like” file format to support use cases where a columnar/tabular format does not make sense. This closely matches Nominal’s streaming API, making it useful for use cases where network connection drops during streaming and a backup file needs to be created.

If this schema is not used, will result in a failed ingestion. { “type”: “record”, “name”: “AvroStream”, “namespace”: “io.nominal.ingest”, “fields”: [ { “name”: “channel”, “type”: “string”, “doc”: “Channel/series name (e.g., ‘vehicle_id’, ‘col_1’, ‘temperature’)”, }, { “name”: “timestamps”, “type”: {“type”: “array”, “items”: “long”}, “doc”: “Array of numeric timestamps; see timestampType for interpretation”, }, { “name”: “values”, “type”: {“type”: “array”, “items”: [“double”, “string”]}, “doc”: “Array of values. Can either be doubles or strings”, }, { “name”: “tags”, “type”: {“type”: “map”, “values”: “string”}, “default”: {}, “doc”: “Key-value metadata tags”, }, ], }

source IngestSource required

​

target DatasetIngestTarget required

​

additionalFileTags map of TagName to string

Specifies a tag set to apply to all data in the file.

timestampType AvroNumericTimestampType

How to interpret the numeric values in the timestamps array. Defaults to epoch-nanoseconds when omitted, matching the original contract of this API.

ChannelConfig#

prefixTreeDelimiter string

If set, will construct a prefix tree for channels of the dataset using the given delimiter.

ContainerizedOpts#

sources map of EnvironmentVariable to IngestSource

​

arguments map of EnvironmentVariable to string

​

extractorRid string (RID) required

​

timestampMetadata TimestampMetadata

​

tag string

​

target DatasetIngestTarget required

​

additionalFileTags map of TagName to string

Specifies a tag set to apply to all data in the file.

CsvOpts#

Options for ingesting csv files. Supported file formats include .csv, .csv.gz

source IngestSource required

​

target DatasetIngestTarget required

​

timestampMetadata TimestampMetadata required

​

channelPrefix string required

​

channelNameOverrides map of ColumnName to string

​

tagKeysFromColumns array of string deprecated

Deprecated: Deprecated in favor of tagColumns.

tagColumns map of TagName to string

A map of tag names to column names to derive the tag values from.

additionalFileTags map of TagName to string

Specifies a tag set to apply to all data in the file.

excludeColumns array of string

A set of column names to exclude from ingestion. These columns will not be ingested as channels. Useful for excluding columns that contain unsupported data types like multidimensional arrays.

CustomTimestamp#

format string required

The format string should be in the format of the DateTimeFormatter class in Java.

defaultYear integer

Default year is accepted as an optional field for cases like IRIG time format and will be overridden by year in time format.

defaultDayOfYear integer

Default day of year is accepted as an optional field for cases like IRIG time format and will be overridden by day of year in time format.

DataflashOpts#

source IngestSource required

​

target DatasetIngestTarget required

​

ignoreInvalidTimestamps boolean

If true, skips dataflash records whose timestamp cannot be represented (corrupted/out-of-range) instead of failing the entire ingest. If not provided, defaults to false.

additionalFileTags map of TagName to string

Specifies a tag set to apply to all data in the file.

DatasetIngestTarget#

Set type to the name of one of the other fields, and set only that field.

new NewDatasetIngestDestination

​

existing ExistingDatasetIngestDestination

​

DeprecatedNewCsv#

source IngestSource required

​

name string

​

properties map of string to string

​

timeColumnSpec TimestampMetadata

​

channelConfig ChannelConfig

​

DeprecatedNewDataSource#

Set type to the name of one of the other fields, and set only that field.

csv DeprecatedNewCsv

​

Duration#

seconds integer (int64) required

​

nanos integer (int64) required

​

Empty#

An empty JSON object, {}.

EpochTimestamp#

timeUnit TimeUnit required

​

ExistingDatasetIngestDestination#

datasetRid string (RID) required

​

expandRunBounds array of string (RID)

The ingest will keep track of the range of timestamps in the ingested data and update the included run’s time bounds. For runs with no end time (uninitialized), it will always set the start time to the file’s start time and set the end time to the file’s end time. For runs with existing end times, it will only expand the bounds (earliest start time, latest end time).

ExistingSpatialIngestDestination#

spatialRid string (RID) required

​

ExistingVideoIngestDestination#

videoRid string (RID) required

RID of the video to ingest the newly created video file to.

videoFileDetails VideoFileIngestDetails

Metadata to associate with any created video file

FileStoreIngestSource#

A resolved File Store revision to ingest, whose bytes are read in place from File Store storage.

fileRevisionRid string (RID) required

​

GcsIngestSource#

path string required

​

ingest.api.NoTimestampManifest#

startingTimestamp UtcTimestamp required

​

scaleParameter ingest.api.ScaleParameter

A field that specifies that the frame rate of the video does not match the frame rate of the camera | i.e. a slowed down or sped up video. Can specify either the camera frame rate or the absolute end time.

ingest.api.ScaleParameter#

Set type to the name of one of the other fields, and set only that field.

trueFrameRate number

​

endingTimestamp UtcTimestamp

the timestamp corresponding to absolute starting timestamp plus absolute duration of the video.

scaleFactor number

the scale factor can be used to calculate whether media duration differs from a video’s | real duration, and if so, the true frame rate of the camera. The video time will thus be scaled | by the ratio of the real duration to media duration, or media frame rate to true frame rate.

IngestDatasetFileDetails#

datasetFileId string (UUID)

​

datasetRid string (RID) required

​

IngestDataSource#

Set type to the name of one of the other fields, and set only that field.

existingDataSource string (RID)

​

newDataSource DeprecatedNewDataSource

​

newDataSourceV2 NewDataSource

​

IngestDetails#

Set type to the name of one of the other fields, and set only that field.

dataset IngestDatasetFileDetails

​

video IngestVideoFileDetails deprecated

Deprecated: Returned only for legacy Video-entity ingests; videoV2 ingests return dataset details carrying the created dataset file id.

spatial IngestSpatialDetails

​

IngestMetadata#

timestampMetadata TimestampMetadata

The timestamp metadata will be recovered from the dataset files if possible. Older datasets may have unrecoverable timestamp metadata. If unrecoverable, reingestion will throw MissingMetadataForReingest if not provided in request.

channelPrefix string

Channel prefix to use when reingesting the dataset. Defaults to empty string. Not recoverable from prior ingests and must be provided in request if needed.

channelNameOverrides map of ColumnName to string

Renames individual columns to explicit channel names. Composes with channelPrefix: final name = channelPrefix + channelNameOverrides.getOrDefault(column, column). Defaults to empty map. Not recoverable from prior ingests and must be provided in request if needed. At most 1000 entries may be provided. Resolved channel names must be unique across columns; renaming a column onto another column’s name fails the ingest.

tagColumns map of TagName to string

A map of tag names to column names to derive the tag values from. Not recoverable from prior ingests and must be provided in request if needed.

additionalFileTags map of TagName to string

Additional tags to apply to all dataset files within the the given dataset. Not recoverable from prior ingests and must be provided in request if needed.

IngestOptions#

Set type to the name of one of the other fields, and set only that field.

dataflash DataflashOpts

​

mcapProtobufTimeseries McapProtobufTimeseriesOpts

​

journalJson JournalJsonOpts

​

csv CsvOpts

​

parquet ParquetOpts

​

video VideoOpts deprecated

Deprecated: Legacy Video-entity ingest. Use videoV2, which ingests video as a channel on a dataset.

videoV2 VideoOptsV2

​

containerized ContainerizedOpts

​

avroStream AvroStreamOpts

​

pointCloud PointCloudOpts

​

IngestRunDataSource#

dataSource IngestDataSource required

​

timeOffsetSpec TimeOffsetSpec

​

IngestSource#

Set type to the name of one of the other fields, and set only that field.

s3 S3IngestSource

​

gcs GcsIngestSource

​

presignedFile PresignedFileIngestSource

​

fileStore FileStoreIngestSource

​

IngestSpatialDetails#

spatialRid string (RID) required

​

IngestVideoFileDetails#

videoFileRid string (RID) required

​

videoRid string (RID) required

​

Iso8601Timestamp#

An empty JSON object, {}.

JournalJsonOpts#

source IngestSource required

​

target DatasetIngestTarget required

​

channel string

If provided, ingests logs to the given channel. By default, log data will be ingested to a channel named ‘logs’.

timestampMetadata JournalTimestampMetadata

Optional timestamp field for the JSON lines. When provided, each line is timestamped from this field instead of the journald default __REALTIME_TIMESTAMP, allowing non-journald JSON logs to ingest. Each line must still contain a MESSAGE field. Only numeric epoch timestamps are supported. When absent, the journald default (__REALTIME_TIMESTAMP, microseconds) is used.

JournalTimestampMetadata#

fieldName string required

​

epochOfTimeUnit EpochTimestamp required

​

LogTime#

An empty JSON object, {}.

McapChannelLocator#

Locator for a channel in an mcap file. Channel name is not guaranteed to be unique, so channel ID should be used for mcap files with duplicate channel names.

Set type to the name of one of the other fields, and set only that field.

topic string

​

id integer

​

McapChannels#

Set type to the name of one of the other fields, and set only that field.

all Empty

​

include array of McapChannelLocator

​

exclude array of McapChannelLocator

​

McapProtobufTimeseriesOpts#

source IngestSource required

​

target DatasetIngestTarget required

​

channelFilter McapChannels required

​

timestampType McapTimestampType required

​

ignoreInvalidTopics boolean

If true, skips invalid MCAP topics during ingestion instead of failing the entire ingest. If not provided, defaults to false.

additionalFileTags map of TagName to string

Specifies a tag set to apply to all data in the file.

McapTimestampManifest#

Timestamps are derived from the mcap file containing the video frames.

mcapChannelLocator McapChannelLocator required

​

McapTimestampType#

LogTime is default timestamp for MCAP messages and should be used in most cases.

Set type to the name of one of the other fields, and set only that field.

logTime LogTime

​

NewDatasetIngestDestination#

datasetName string

​

datasetDescription string

​

properties map of PropertyName to string

​

labels array of string

​

channelConfig ChannelConfig

​

workspace string (RID)

The workspace in which to create the dataset. If not provided, the dataset will be created in the default workspace for the user’s organization, if the default workspace for the organization is configured.

markingRids array of string (RID)

The markings to apply to the created dataset. If not provided, the dataset will be visible to all users in the same workspace.

NewDataSource#

source IngestSource required

​

properties map of PropertyName to string

​

labels array of string

​

description string

​

name string

​

timeColumnSpec TimestampMetadata

​

channelConfig ChannelConfig

​

NewVideoIngestDestination#

title string

Title of the Video that will get created. If not provided, a name is deduced from the ingested file.

description string

Description that is applied to the newly created video

properties map of PropertyName to string

Key-Value properties that are applied to the newly created video

labels array of string

Labels that are applied to the newly created video

videoFileDetails VideoFileIngestDetails

Metadata to associate with any created video file

workspace string (RID)

The workspace in which to create the video. If not provided, the video will be created in the default workspace for the user’s organization, if the default workspace for the organization is configured.

markingRids array of string (RID)

The markings to apply to the created video. If not provided, the video will be visible to all users in the same workspace.

ParquetOpts#

Options for ingesting parquet files. Supported file formats include .parquet, .parquet.gz and archives such as .tar, .tar.gz, and .zip (must set the isArchive flag).

source IngestSource required

​

target DatasetIngestTarget required

​

timestampMetadata TimestampMetadata required

​

channelPrefix string required

​

channelNameOverrides map of ColumnName to string

​

tagKeysFromColumns array of string deprecated

Deprecated: Deprecated in favor of tagColumns.

tagColumns map of TagName to string

A map of tag names to column names to derive the tag values from.

additionalFileTags map of TagName to string

Specifies a tag set to apply to all data in the file.

isArchive boolean

If true, the file is an archive. Supported archive formats include .tar, .tar.gz, and .zip. Only files ending in .parquet within the archive will be ingested. If field not provided, defaults to false.

excludeColumns array of string

A set of column names to exclude from ingestion. These columns will not be ingested as channels. Useful for excluding columns that contain unsupported data types like multidimensional arrays.

PointCloudIngestTarget#

Set type to the name of one of the other fields, and set only that field.

existing ExistingSpatialIngestDestination

​

PointCloudMetadata#

Metadata about the sensor that captured a point cloud.

sensorModel string

The sensor model name, e.g. “Velodyne VLP-16”, “Ouster OS1-128”.

coordinateSystem string

Coordinate reference system identifier, e.g. “EPSG:4326”, “ENU”.

resolutionMm number

Spatial resolution of the point cloud in millimeters.

scanPattern ScanPattern

The scan pattern of the lidar sensor.

PointCloudOpts#

Options for ingesting point cloud data into Dagger via Nominal.

source IngestSource required

​

target PointCloudIngestTarget required

​

daggerImportConfig any JSON value required

The Dagger **v2** ImportRequest body — everything from that schema except source_uri, which Nominal fills in from the resolved presigned URL for source. Forwarded to Dagger’s POST /v2/imports/{model_uuid}. Construct via the language’s generated dagger-client types (e.g. dagger_client.models.ImportRequest in Python). Carries the required archetype (attribute definitions and sampler configuration), an optional format (the source format and its format-specific parameters, including the geometry_type and columns a CSV source needs; omit it to let Dagger infer the format from the file extension), and an optional quiche_config — all of which are domain knowledge the client owns when it produced the file. The legacy v1 shape, with geometry_type and columns at the top level, is rejected.

sensorMetadata PointCloudMetadata

Sensor metadata to associate with the point cloud.

channel string required

Channel name for workbook integration.

tags map of TagName to string

Tags to apply to the point cloud series.

PresignedFileIngestSource#

url string required

​

ReingestFilesMapping#

Maps the source dataset file ID to the destination dataset and file ID.

fileMap map of DatasetFileId to IngestDetails

​

RelativeTimestamp#

timeUnit TimeUnit required

​

offset string (date-time)

Starting timestamp to use when indexing the file. This field is required when uploading a new file to an existing dataset.

S3IngestSource#

path string required

​

ScanPattern#

ROTATING

​

SOLID_STATE

​

FLASH

​

MECHANICAL

​

TimeOffsetSpec#

Set type to the name of one of the other fields, and set only that field.

nanos Duration

​

Timestamp#

Deprecated for new APIs. Use google.protobuf.Timestamp for absolute instants in proto APIs.

This legacy nanosecond precision timestamp type is represented by an epoch time in seconds and a nanosecond offset. The nanosecond offset is from the start of the epoch second, so must be less than 1 billion.

seconds integer (int64) required

​

nanos integer (int64) required

​

TimestampManifest#

The timestamp manifest files will contain a list of absolute timestamps, in nanoseconds, that correspond to each frame in a video. Each file should be of type JSON and store a single list, the length of which equals the number of frames in its corresponding video.

sources array of IngestSource

​

TimestampMetadata#

seriesName string required

​

timestampType TimestampType required

​

TimestampType#

Set type to the name of one of the other fields, and set only that field.

relative RelativeTimestamp

​

absolute AbsoluteTimestamp

​

TimeUnit#

DAYS

​

HOURS

​

MINUTES

​

SECONDS

​

MILLISECONDS

​

MICROSECONDS

​

NANOSECONDS

​

UtcTimestamp#

secondsSinceEpoch integer (int64) required

​

offsetNanoseconds integer (int64)

​

video.api.NoTimestampManifest#

these values will not be updated after ingest time, to allow for resetting. The updated values are stored implicitly through the segment timestamps.

startingTimestamp Timestamp required

Specifies the original starting timestamp of the video.

scaleParameter video.api.ScaleParameter

A field that specifies that the frame rate of the video does not match the frame rate of the camera | i.e. a slowed down or sped up video. Can specify either the camera frame rate or the absolute end time.

video.api.ScaleParameter#

Set type to the name of one of the other fields, and set only that field.

trueFrameRate number

​

endingTimestamp Timestamp

​

scaleFactor number

the scale factor can be used to calculate whether media duration differs from a video’s | real duration, and if so, the true frame rate of the camera. The video time will thus be scaled | by the ratio of the real duration to media duration, or media frame rate to true frame rate.

VideoFileIngestDetails#

fileDescription string

Description that is applied to the newly created video file. If not provided, defaults to the description used to create the video.

fileProperties map of PropertyName to string deprecated

Key-Value properties that are applied to the newly created video file. Deprecated: Field is ignored as video files do not have properties. Will be removed after 4/5/2025

fileLabels array of string deprecated

Labels that are applied to the newly created video file. Deprecated: Field is ignored as video files do not have labels. Will be removed after 4/5/2025

VideoFileTimestampManifest#

A video file’s timestamp manifest specifies how to determine absolute timestamps for each frame. In an MCAP file, this data is embedded within the source file. Alternatively, the manifest may be provided as an external sidecar file. If no manifest is available, the timestamps are calculated by applying a starting offset to the presentation timestamp of each frame.

Set type to the name of one of the other fields, and set only that field.

mcap McapTimestampManifest

​

s3path string

​

s3paths array of string

Multiple sidecar manifest files whose per-frame timestamps are concatenated in list order. Used when a video’s frame timestamps are recorded across several files.

noManifest video.api.NoTimestampManifest

​

VideoIngestTarget#

Set type to the name of one of the other fields, and set only that field.

new NewVideoIngestDestination

​

existing ExistingVideoIngestDestination

​

VideoOpts#

source IngestSource required

​

target VideoIngestTarget required

​

timestampManifest VideoFileTimestampManifest required

​

overWriteSegments boolean

If true, overlapping segments from other video files within the same video will be deleted before inserting new segments. The cached segment metadata for affected files will be recomputed.

VideoOptsV2#

source IngestSource required

​

target DatasetIngestTarget required

​

timestampManifest VideoFileTimestampManifest required

​

channel string required

​

tags map of TagName to string

​

overWriteSegments boolean

If true, overlapping segments from other dataset files within the same series will be deleted before inserting new segments. The cached segment metadata for affected files will be recomputed.

VideoTimestampManifest#

Set type to the name of one of the other fields, and set only that field.

noManifest ingest.api.NoTimestampManifest

​

timestampManifests TimestampManifest

​

Page design based on Scalar, MIT License.