Ingest service (gRPC)#

Starts V2 ingest jobs and returns the RID used to track asynchronous progress.

Endpoints#

Ingest#

gRPC /nominal.ingest.v2.IngestService/Ingest

Submit a V2 ingest job.

Request: IngestRequest

dataset_rid string required

Existing dataset that all items in this job will ingest into.

runs_to_expand array of string

​

items array of IngestItem

Units of ingest work to submit under one ingest job RID. At least 1 item.

Fields of IngestItem
tags map of string to string

Tags applied only to this ingest item. These override request-level tags on conflict.

one of 7 fields required

​

file FileIngestItem

​

Fields of FileIngestItem
source IngestSource required

​

Fields of IngestSource
one of s3, gcs, presigned_file, file_store required

​

s3 S3IngestSource

​

Fields of S3IngestSource
path string required

​

gcs GcsIngestSource

​

Fields of GcsIngestSource
path string required

​

presigned_file PresignedFileIngestSource

​

Fields of PresignedFileIngestSource
url string required

​

file_store FileStoreIngestSource

Initially supported for direct file ingest only.

Fields of FileStoreIngestSource
file_revision_rid string

Resolved File Store content target to ingest.

ingest FileIngestOptions required

​

Fields of FileIngestOptions
timestamp_metadata ingest.v2.TimestampMetadata required

​

Fields of ingest.v2.TimestampMetadata
column string required

​

type TimestampType required

​

Fields of TimestampType
one of relative, absolute required

​

relative RelativeTimestamp

​

Fields of RelativeTimestamp
time_unit string

Unit in which relative offsets are expressed. Aliases the classic REST TimeUnit alias.

offset string (date-time)

Starting timestamp to use when indexing the file.

absolute AbsoluteTimestamp

​

Fields of AbsoluteTimestamp
one of iso8601, epoch_of_time_unit, custom_format required

​

iso8601 Iso8601Timestamp

Request

import grpc
from nominal.protos.ingest.v2 import ingest_service_pb2, ingest_service_pb2_grpc

channel = grpc.secure_channel("api.gov.nominal.io:443", grpc.ssl_channel_credentials())
ingest_service = ingest_service_pb2_grpc.IngestServiceStub(channel)
auth = [("authorization", "Bearer <api key>")]

request = ingest_service_pb2.IngestRequest(dataset_rid="<dataset rid>")
response = ingest_service.Ingest(request, metadata=auth)
print(response)

package main

import (
	"context"
	"crypto/tls"
	"fmt"
	"log"

	ingestv2 "github.com/nominal-io/nominal-api-protos-go/nominal/protos/ingest/v2"
	"google.golang.org/grpc"
	"google.golang.org/grpc/credentials"
	"google.golang.org/grpc/metadata"
)

func main() {
	conn, err := grpc.NewClient("api.gov.nominal.io:443",
		grpc.WithTransportCredentials(credentials.NewTLS(&tls.Config{})))
	if err != nil {
		log.Fatal(err)
	}
	defer conn.Close()
	ingestService := ingestv2.NewIngestServiceClient(conn)
	ctx := metadata.AppendToOutgoingContext(context.Background(), "authorization", "Bearer <api key>")

	request := &ingestv2.IngestRequest{
		DatasetRid: "<dataset rid>",
	}
	response, err := ingestService.Ingest(ctx, request)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(response)
}

Response

{
  "ingest_job_rid": "<ingest job rid>"
}

Types#

AbsoluteTimestamp#

Absolute timestamp parser variants.

Set exactly one of the fields below.

one of iso8601, epoch_of_time_unit, custom_format required

​

iso8601 Iso8601Timestamp

​

epoch_of_time_unit EpochTimestamp

​

custom_format CustomTimestamp

​

AvroIngestOptions#

Avro ingest options.

A message with no fields.

ContainerizedIngestItem#

One containerized extractor invocation. Input sources are bound to environment variables.

extractor_rid string required

​

sources map of string to IngestSource

​

arguments map of string to string

​

timestamp_metadata ingest.v2.TimestampMetadata

​

CsvFormat#

Set exactly one of the fields below.

one of wide, long required

​

wide WideFormat

​

long LongFormat

​

CsvIngestOptions#

CSV ingest options.

data_row must be greater than header_row. units_row must not be header_row. units_row must be less than data_row, which defaults to header_row + 1.

format CsvFormat required

​

header_row integer

One-based record number of the header, defaulting to 1; must be positive. Records are counted from 1, a multiline record counts as one, and blank lines are ignored. Greater than 0.

data_row integer

One-based record number of the first data record, defaulting to header_row + 1; must be greater than header_row. Records that fall between header_row and data_row are skipped. Greater than 0.

units_row integer

One-based record number holding each column’s unit; must not be header_row and must precede data_row, so reading a units row requires setting data_row past it. Cells match columns by position, so the record needs a placeholder cell for the timestamp column, and a column whose cell is empty gets no unit. Absent means no units row. FileIngestOptions.units overrides a unit read here, per channel. Greater than 0.

CustomTimestamp#

Absolute timestamps parsed with a user-supplied format string.

format string

The format string should be in the format of the DateTimeFormatter class in Java.

default_year integer

Default year is accepted as an optional field for cases like IRIG time format. Will be overridden by year in time format.

default_day_of_year integer

Default day of year is accepted as an optional field for cases like IRIG time format. Will be overridden by day of year in time format. From 1 to 366.

DataflashIngestItem#

One ArduPilot dataflash log conversion. Worker-mcap converts the log into an avro stream that is then ingested into the dataset.

source IngestSource required

​

EpochTimestamp#

Absolute timestamps expressed as an integer count of some time unit since the Unix epoch.

time_unit string

Unit in which epoch values are expressed (e.g. SECONDS, MILLISECONDS).

FileIngestItem#

Direct file ingest work.

source IngestSource required

​

ingest FileIngestOptions required

​

FileIngestOptions#

Direct file ingest mode.

Set exactly one of csv, parquet, avro.

timestamp_metadata ingest.v2.TimestampMetadata required

​

units map of string to string

Unit per channel. Overrides the unit read from CsvIngestOptions.units_row for the channels named here; channels absent from this map keep their units-row unit.

channel_prefix string

​

channel_name_overrides map of string to string

Renames individual columns to explicit channel names. Composes with channel_prefix: final name = channel_prefix + channel_name_overrides.getOrDefault(column, column). At most 1000 entries may be provided. Resolved channel names must be unique across columns; renaming a column onto another column’s name fails the ingest.

one of csv, parquet, avro required

​

csv CsvIngestOptions

​

parquet ParquetIngestOptions

​

avro AvroIngestOptions

​

FileStoreIngestSource#

File Store source. The resolved file RID supplies provider-specific pinning and dereference metadata.

file_revision_rid string

Resolved File Store content target to ingest.

GcsIngestSource#

path string required

​

ingest.v2.TimestampMetadata#

Timestamp metadata for ingest items that derive timestamps from an input column or field.

column string required

​

type TimestampType required

​

IngestItem#

One unit of ingest work submitted under a single ingest job.

Set exactly one of file, log, containerized, video, point_cloud, mcap, dataflash.

tags map of string to string

Tags applied only to this ingest item. These override request-level tags on conflict.

one of 7 fields required

​

file FileIngestItem

​

log LogIngestItem

​

containerized ContainerizedIngestItem

​

video VideoIngestItem

​

point_cloud PointCloudIngestItem

​

mcap McapIngestItem

​

dataflash DataflashIngestItem

​

IngestSource#

Source file location.

Set exactly one of the fields below.

one of s3, gcs, presigned_file, file_store required

​

s3 S3IngestSource

​

gcs GcsIngestSource

​

presigned_file PresignedFileIngestSource

​

file_store FileStoreIngestSource

Initially supported for direct file ingest only.

Iso8601Timestamp#

Absolute timestamps serialized in ISO-8601 form. No additional parser configuration is needed.

A message with no fields.

LogIngestItem#

Single-channel log or journal ingest work.

source IngestSource required

​

channel string

​

timestamp_metadata ingest.v2.TimestampMetadata

​

message_field string

JSON field whose value becomes the log message. Defaults to the journald MESSAGE field.

LongFormat#

Long format: each row or record contains one channel, timestamp, and value sample.

channel_column string

Defaults to “channel”.

value_column string

Defaults to “value”.

tags_column string

Defaults to “tags”.

McapChannelLocator#

Locates the video’s channel within an mcap file. Topics are not guaranteed unique, so an mcap file with duplicate topics must be located by id.

Set exactly one of the fields below.

one of topic, id required

​

topic string

​

id integer

​

McapChannelSelection#

Selects mcap channels by topic name. All channels are ingested when neither list is set.

Set at most one of the fields below.

one of include_topics, exclude_topics

​

include_topics McapTopicNames

​

exclude_topics McapTopicNames

​

McapIngestItem#

One mcap file conversion. Worker-mcap converts the file’s protobuf channels into an avro stream that is then ingested into the dataset.

source IngestSource required

​

channels McapChannelSelection

Which mcap channels to ingest, selected by topic name. All channels when unset.

ignore_invalid_topics boolean

Skip topics whose schemas cannot be decoded instead of failing the conversion.

McapTimestampManifest#

Frame timestamps are embedded in the mcap file that carries the video frames, rather than supplied as a sidecar manifest.

channel_locator McapChannelLocator required

​

McapTopicNames#

topics array of string

​

NoTimestampManifest#

starting_timestamp string (date-time) required

​

scale_parameter ScaleParameter

​

ParquetFormat#

Set exactly one of the fields below.

one of wide, long required

​

wide WideFormat

​

long LongFormat

​

ParquetIngestOptions#

Parquet ingest options.

format ParquetFormat required

​

is_archive boolean

​

PointCloudIngestItem#

Dataset-backed point cloud ingest work.

source IngestSource required

​

ingest PointCloudIngestOptions required

​

PointCloudIngestOptions#

sensor_metadata PointCloudSensorMetadata

​

channel string required

​

PointCloudSensorMetadata#

Placeholder for point-cloud sensor metadata until the spatial metadata proto is promoted.

properties map of string to string

​

PresignedFileIngestSource#

url string required

​

RelativeTimestamp#

Relative timestamp parser: interprets each row’s timestamp as an offset from a fixed origin.

time_unit string

Unit in which relative offsets are expressed. Aliases the classic REST TimeUnit alias.

offset string (date-time)

Starting timestamp to use when indexing the file.

S3IngestSource#

path string required

​

ScaleParameter#

Set exactly one of the fields below.

one of true_frame_rate, ending_timestamp, scale_factor required

​

true_frame_rate number

​

ending_timestamp string (date-time)

​

scale_factor number

​

TimestampManifestFiles#

Sidecar manifest files whose per-frame timestamps are concatenated in list order. Used when a video’s frame timestamps are recorded across several files.

sources array of IngestSource

At least 1 item.

TimestampType#

Selects between relative-offset and absolute-timestamp parsing strategies.

Set exactly one of the fields below.

one of relative, absolute required

​

relative RelativeTimestamp

​

absolute AbsoluteTimestamp

​

VideoIngestItem#

Dataset-file-backed video ingest work.

source IngestSource required

​

ingest VideoIngestOptions required

​

additional_sources array of IngestSource

Videos appended in order after source, ingested as one continuous recording on the channel. Each becomes its own dataset file. The first file starts at the no-manifest starting timestamp and each later file starts where the previous one ends. Requires a no_manifest timestamp manifest and sources other than File Store; at most 99, so 100 files in all. At most 99 items.

VideoIngestOptions#

timestamp_manifest VideoTimestampManifest required

​

channel string required

​

overwrite_segments boolean

​

VideoTimestampManifest#

Set exactly one of the fields below.

one of no_manifest, timestamp_manifest_files, mcap required

​

no_manifest NoTimestampManifest

​

timestamp_manifest_files TimestampManifestFiles

​

mcap McapTimestampManifest

​

WideFormat#

Wide format: each row or record contains one timestamp and many channels as columns.

tag_columns map of string to string

​

exclude_columns array of string

​

Page design based on Scalar, MIT License.