Ingest service#

Starts V2 ingest jobs and returns the RID used to track asynchronous progress.

OpenAPI file: nominal.ingest.v2.json

Endpoints#

Ingest#

POST /ingest/v2

Submit a V2 ingest job.

Body

datasetRid string required

Existing dataset that all items in this job will ingest into.

runsToExpand array of string

​

items array of IngestItem

Units of ingest work to submit under one ingest job RID. At least 1 item.

Fields of IngestItem
tags map of string to string

Tags applied only to this ingest item. These override request-level tags on conflict.

one of 7 fields required

​

file FileIngestItem

​

Fields of FileIngestItem
source IngestSource required

​

Fields of IngestSource
one of s3, gcs, presignedFile, fileStore required

​

s3 S3IngestSource

​

Fields of S3IngestSource
path string required

​

gcs GcsIngestSource

​

Fields of GcsIngestSource
path string required

​

presignedFile PresignedFileIngestSource

​

Fields of PresignedFileIngestSource
url string required

​

fileStore FileStoreIngestSource

Initially supported for direct file ingest only.

Fields of FileStoreIngestSource
fileRevisionRid string

Resolved File Store content target to ingest.

ingest FileIngestOptions required

​

Fields of FileIngestOptions
timestampMetadata ingest.v2.TimestampMetadata required

​

Fields of ingest.v2.TimestampMetadata
column string required

​

type TimestampType required

​

Fields of TimestampType
one of relative, absolute required

​

relative RelativeTimestamp

​

Fields of RelativeTimestamp
timeUnit string

Unit in which relative offsets are expressed. Aliases the classic REST TimeUnit alias.

offset string (date-time)

Starting timestamp to use when indexing the file.

absolute AbsoluteTimestamp

​

Fields of AbsoluteTimestamp
one of iso8601, epochOfTimeUnit, customFormat required

​

iso8601 Iso8601Timestamp

Request

curl -X POST "https://api.gov.nominal.io/api/ingest/v2" \
  -H "Authorization: Bearer <api key>" \
  -H "Content-Type: application/json" \
  --data '{
  "datasetRid": "<dataset rid>"
}'

import requests

response = requests.post(
    "https://api.gov.nominal.io/api/ingest/v2",
    headers={"Authorization": "Bearer <api key>"},
    json={"datasetRid": "<dataset rid>"},
)
response.raise_for_status()
print(response.json())

import httpx

response = httpx.post(
    "https://api.gov.nominal.io/api/ingest/v2",
    headers={"Authorization": "Bearer <api key>"},
    json={"datasetRid": "<dataset rid>"},
)
response.raise_for_status()
print(response.json())

import asyncio

import aiohttp


async def main() -> None:
    async with aiohttp.ClientSession() as session:
        async with session.post(
            "https://api.gov.nominal.io/api/ingest/v2",
            headers={"Authorization": "Bearer <api key>"},
            json={"datasetRid": "<dataset rid>"},
        ) as response:
            response.raise_for_status()
            print(await response.json())


asyncio.run(main())

Response: 200

{
  "ingestJobRid": "<ingest job rid>"
}

Types#

AbsoluteTimestamp#

Absolute timestamp parser variants.

Set exactly one of the fields below.

one of iso8601, epochOfTimeUnit, customFormat required

​

iso8601 Iso8601Timestamp

​

epochOfTimeUnit EpochTimestamp

​

customFormat CustomTimestamp

​

AvroIngestOptions#

Avro ingest options.

An empty JSON object, {}.

ContainerizedIngestItem#

One containerized extractor invocation. Input sources are bound to environment variables.

extractorRid string required

​

sources map of string to IngestSource

​

arguments map of string to string

​

timestampMetadata ingest.v2.TimestampMetadata

​

CsvFormat#

Set exactly one of the fields below.

one of wide, long required

​

wide WideFormat

​

long LongFormat

​

CsvIngestOptions#

CSV ingest options.

data_row must be greater than header_row. units_row must not be header_row. units_row must be less than data_row, which defaults to header_row + 1.

format CsvFormat required

​

headerRow integer

One-based record number of the header, defaulting to 1; must be positive. Records are counted from 1, a multiline record counts as one, and blank lines are ignored. Greater than 0.

dataRow integer

One-based record number of the first data record, defaulting to header_row + 1; must be greater than header_row. Records that fall between header_row and data_row are skipped. Greater than 0.

unitsRow integer

One-based record number holding each column’s unit; must not be header_row and must precede data_row, so reading a units row requires setting data_row past it. Cells match columns by position, so the record needs a placeholder cell for the timestamp column, and a column whose cell is empty gets no unit. Absent means no units row. FileIngestOptions.units overrides a unit read here, per channel. Greater than 0.

CustomTimestamp#

Absolute timestamps parsed with a user-supplied format string.

format string

The format string should be in the format of the DateTimeFormatter class in Java.

defaultYear integer

Default year is accepted as an optional field for cases like IRIG time format. Will be overridden by year in time format.

defaultDayOfYear integer

Default day of year is accepted as an optional field for cases like IRIG time format. Will be overridden by day of year in time format. From 1 to 366.

DataflashIngestItem#

One ArduPilot dataflash log conversion. Worker-mcap converts the log into an avro stream that is then ingested into the dataset.

source IngestSource required

​

EpochTimestamp#

Absolute timestamps expressed as an integer count of some time unit since the Unix epoch.

timeUnit string

Unit in which epoch values are expressed (e.g. SECONDS, MILLISECONDS).

FileIngestItem#

Direct file ingest work.

source IngestSource required

​

ingest FileIngestOptions required

​

FileIngestOptions#

Direct file ingest mode.

Set exactly one of csv, parquet, avro.

timestampMetadata ingest.v2.TimestampMetadata required

​

units map of string to string

Unit per channel. Overrides the unit read from CsvIngestOptions.units_row for the channels named here; channels absent from this map keep their units-row unit.

channelPrefix string

​

channelNameOverrides map of string to string

Renames individual columns to explicit channel names. Composes with channel_prefix: final name = channel_prefix + channel_name_overrides.getOrDefault(column, column). At most 1000 entries may be provided. Resolved channel names must be unique across columns; renaming a column onto another column’s name fails the ingest.

one of csv, parquet, avro required

​

csv CsvIngestOptions

​

parquet ParquetIngestOptions

​

avro AvroIngestOptions

​

FileStoreIngestSource#

File Store source. The resolved file RID supplies provider-specific pinning and dereference metadata.

fileRevisionRid string

Resolved File Store content target to ingest.

GcsIngestSource#

path string required

​

ingest.v2.TimestampMetadata#

Timestamp metadata for ingest items that derive timestamps from an input column or field.

column string required

​

type TimestampType required

​

IngestItem#

One unit of ingest work submitted under a single ingest job.

Set exactly one of file, log, containerized, video, pointCloud, mcap, dataflash.

tags map of string to string

Tags applied only to this ingest item. These override request-level tags on conflict.

one of 7 fields required

​

file FileIngestItem

​

log LogIngestItem

​

containerized ContainerizedIngestItem

​

video VideoIngestItem

​

pointCloud PointCloudIngestItem

​

mcap McapIngestItem

​

dataflash DataflashIngestItem

​

IngestSource#

Source file location.

Set exactly one of the fields below.

one of s3, gcs, presignedFile, fileStore required

​

s3 S3IngestSource

​

gcs GcsIngestSource

​

presignedFile PresignedFileIngestSource

​

fileStore FileStoreIngestSource

Initially supported for direct file ingest only.

Iso8601Timestamp#

Absolute timestamps serialized in ISO-8601 form. No additional parser configuration is needed.

An empty JSON object, {}.

LogIngestItem#

Single-channel log or journal ingest work.

source IngestSource required

​

channel string

​

timestampMetadata ingest.v2.TimestampMetadata

​

messageField string

JSON field whose value becomes the log message. Defaults to the journald MESSAGE field.

LongFormat#

Long format: each row or record contains one channel, timestamp, and value sample.

channelColumn string

Defaults to “channel”.

valueColumn string

Defaults to “value”.

tagsColumn string

Defaults to “tags”.

McapChannelLocator#

Locates the video’s channel within an mcap file. Topics are not guaranteed unique, so an mcap file with duplicate topics must be located by id.

Set exactly one of the fields below.

one of topic, id required

​

topic string

​

id integer

​

McapChannelSelection#

Selects mcap channels by topic name. All channels are ingested when neither list is set.

Set at most one of the fields below.

one of includeTopics, excludeTopics

​

includeTopics McapTopicNames

​

excludeTopics McapTopicNames

​

McapIngestItem#

One mcap file conversion. Worker-mcap converts the file’s protobuf channels into an avro stream that is then ingested into the dataset.

source IngestSource required

​

channels McapChannelSelection

Which mcap channels to ingest, selected by topic name. All channels when unset.

ignoreInvalidTopics boolean

Skip topics whose schemas cannot be decoded instead of failing the conversion.

McapTimestampManifest#

Frame timestamps are embedded in the mcap file that carries the video frames, rather than supplied as a sidecar manifest.

channelLocator McapChannelLocator required

​

McapTopicNames#

topics array of string

​

NoTimestampManifest#

startingTimestamp string (date-time) required

​

scaleParameter ScaleParameter

​

ParquetFormat#

Set exactly one of the fields below.

one of wide, long required

​

wide WideFormat

​

long LongFormat

​

ParquetIngestOptions#

Parquet ingest options.

format ParquetFormat required

​

isArchive boolean

​

PointCloudIngestItem#

Dataset-backed point cloud ingest work.

source IngestSource required

​

ingest PointCloudIngestOptions required

​

PointCloudIngestOptions#

sensorMetadata PointCloudSensorMetadata

​

channel string required

​

PointCloudSensorMetadata#

Placeholder for point-cloud sensor metadata until the spatial metadata proto is promoted.

properties map of string to string

​

PresignedFileIngestSource#

url string required

​

RelativeTimestamp#

Relative timestamp parser: interprets each row’s timestamp as an offset from a fixed origin.

timeUnit string

Unit in which relative offsets are expressed. Aliases the classic REST TimeUnit alias.

offset string (date-time)

Starting timestamp to use when indexing the file.

S3IngestSource#

path string required

​

ScaleParameter#

Set exactly one of the fields below.

one of trueFrameRate, endingTimestamp, scaleFactor required

​

trueFrameRate number

​

endingTimestamp string (date-time)

​

scaleFactor number

​

TimestampManifestFiles#

Sidecar manifest files whose per-frame timestamps are concatenated in list order. Used when a video’s frame timestamps are recorded across several files.

sources array of IngestSource

At least 1 item.

TimestampType#

Selects between relative-offset and absolute-timestamp parsing strategies.

Set exactly one of the fields below.

one of relative, absolute required

​

relative RelativeTimestamp

​

absolute AbsoluteTimestamp

​

VideoIngestItem#

Dataset-file-backed video ingest work.

source IngestSource required

​

ingest VideoIngestOptions required

​

additionalSources array of IngestSource

Videos appended in order after source, ingested as one continuous recording on the channel. Each becomes its own dataset file. The first file starts at the no-manifest starting timestamp and each later file starts where the previous one ends. Requires a no_manifest timestamp manifest and sources other than File Store; at most 99, so 100 files in all. At most 99 items.

VideoIngestOptions#

timestampManifest VideoTimestampManifest required

​

channel string required

​

overwriteSegments boolean

​

VideoTimestampManifest#

Set exactly one of the fields below.

one of noManifest, timestampManifestFiles, mcap required

​

noManifest NoTimestampManifest

​

timestampManifestFiles TimestampManifestFiles

​

mcap McapTimestampManifest

​

WideFormat#

Wide format: each row or record contains one timestamp and many channels as columns.

tagColumns map of string to string

​

excludeColumns array of string

​

Page design based on Scalar, MIT License.