# Ingest service (gRPC)

Starts V2 ingest jobs and returns the RID used to track asynchronous progress.

## Endpoints

### Ingest

gRPC `/nominal.ingest.v2.IngestService/Ingest`

Submit a V2 ingest job.

Request: `IngestRequest`

`dataset_rid` string required
:   Existing dataset that all items in this job will ingest into.

`runs_to_expand` array of string
:   ​

`items` array of [`IngestItem`](https://dev.nominal.io/api/grpc/ingest-service.md#ingestitem)
:   Units of ingest work to submit under one ingest job RID. At least 1 item.

    Fields of `IngestItem`

    `tags` map of string to string
    :   Tags applied only to this ingest item. These override request-level tags on conflict.

    one of 7 fields required
    :   ​

        **file**

        `file` [`FileIngestItem`](https://dev.nominal.io/api/grpc/ingest-service.md#fileingestitem)
        :   ​

            Fields of `FileIngestItem`

            `source` [`IngestSource`](https://dev.nominal.io/api/grpc/ingest-service.md#ingestsource) required
            :   ​

                Fields of `IngestSource`

                one of `s3`, `gcs`, `presigned_file`, `file_store` required
                :   ​

                    **s3**

                    `s3` [`S3IngestSource`](https://dev.nominal.io/api/grpc/ingest-service.md#s3ingestsource)
                    :   ​

                        Fields of `S3IngestSource`

                        `path` string required
                        :   ​

                    **gcs**

                    `gcs` [`GcsIngestSource`](https://dev.nominal.io/api/grpc/ingest-service.md#gcsingestsource)
                    :   ​

                        Fields of `GcsIngestSource`

                        `path` string required
                        :   ​

                    **presigned_file**

                    `presigned_file` [`PresignedFileIngestSource`](https://dev.nominal.io/api/grpc/ingest-service.md#presignedfileingestsource)
                    :   ​

                        Fields of `PresignedFileIngestSource`

                        `url` string required
                        :   ​

                    **file_store**

                    `file_store` [`FileStoreIngestSource`](https://dev.nominal.io/api/grpc/ingest-service.md#filestoreingestsource)
                    :   Initially supported for direct file ingest only.

                        Fields of `FileStoreIngestSource`

                        `file_revision_rid` string
                        :   Resolved File Store content target to ingest.

            `ingest` [`FileIngestOptions`](https://dev.nominal.io/api/grpc/ingest-service.md#fileingestoptions) required
            :   ​

                Fields of `FileIngestOptions`

                `timestamp_metadata` [`ingest.v2.TimestampMetadata`](https://dev.nominal.io/api/grpc/ingest-service.md#ingest-v2-timestampmetadata) required
                :   ​

                    Fields of `ingest.v2.TimestampMetadata`

                    `column` string required
                    :   ​

                    `type` [`TimestampType`](https://dev.nominal.io/api/grpc/ingest-service.md#timestamptype) required
                    :   ​

                        Fields of `TimestampType`

                        one of `relative`, `absolute` required
                        :   ​

                            **relative**

                            `relative` [`RelativeTimestamp`](https://dev.nominal.io/api/grpc/ingest-service.md#relativetimestamp)
                            :   ​

                                Fields of `RelativeTimestamp`

                                `time_unit` string
                                :   Unit in which relative offsets are expressed. Aliases the classic REST TimeUnit alias.

                                `offset` string (date-time)
                                :   Starting timestamp to use when indexing the file.

                            **absolute**

                            `absolute` [`AbsoluteTimestamp`](https://dev.nominal.io/api/grpc/ingest-service.md#absolutetimestamp)
                            :   ​

                                Fields of `AbsoluteTimestamp`

                                one of `iso8601`, `epoch_of_time_unit`, `custom_format` required
                                :   ​

                                    **iso8601**

                                    `iso8601` [`Iso8601Timestamp`](https://dev.nominal.io/api/grpc/ingest-service.md#iso8601timestamp)

Request

**Python**

```python
import grpc
from nominal.protos.ingest.v2 import ingest_service_pb2, ingest_service_pb2_grpc

channel = grpc.secure_channel("api.gov.nominal.io:443", grpc.ssl_channel_credentials())
ingest_service = ingest_service_pb2_grpc.IngestServiceStub(channel)
auth = [("authorization", "Bearer <api key>")]

request = ingest_service_pb2.IngestRequest(dataset_rid="<dataset rid>")
response = ingest_service.Ingest(request, metadata=auth)
print(response)
```

**Go**

```go
package main

import (
	"context"
	"crypto/tls"
	"fmt"
	"log"

	ingestv2 "github.com/nominal-io/nominal-api-protos-go/nominal/protos/ingest/v2"
	"google.golang.org/grpc"
	"google.golang.org/grpc/credentials"
	"google.golang.org/grpc/metadata"
)

func main() {
	conn, err := grpc.NewClient("api.gov.nominal.io:443",
		grpc.WithTransportCredentials(credentials.NewTLS(&tls.Config{})))
	if err != nil {
		log.Fatal(err)
	}
	defer conn.Close()
	ingestService := ingestv2.NewIngestServiceClient(conn)
	ctx := metadata.AppendToOutgoingContext(context.Background(), "authorization", "Bearer <api key>")

	request := &ingestv2.IngestRequest{
		DatasetRid: "<dataset rid>",
	}
	response, err := ingestService.Ingest(ctx, request)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(response)
}
```

Response

```json
{
  "ingest_job_rid": "<ingest job rid>"
}
```

## Types

### AbsoluteTimestamp

Absolute timestamp parser variants.

Set exactly one of the fields below.

one of `iso8601`, `epoch_of_time_unit`, `custom_format` required
:   ​

    **iso8601**

    `iso8601` [`Iso8601Timestamp`](https://dev.nominal.io/api/grpc/ingest-service.md#iso8601timestamp)
    :   ​

    **epoch_of_time_unit**

    `epoch_of_time_unit` [`EpochTimestamp`](https://dev.nominal.io/api/grpc/ingest-service.md#epochtimestamp)
    :   ​

    **custom_format**

    `custom_format` [`CustomTimestamp`](https://dev.nominal.io/api/grpc/ingest-service.md#customtimestamp)
    :   ​

### AvroIngestOptions

Avro ingest options.

A message with no fields.

### ContainerizedIngestItem

One containerized extractor invocation. Input sources are bound to environment variables.

`extractor_rid` string required
:   ​

`sources` map of string to [`IngestSource`](https://dev.nominal.io/api/grpc/ingest-service.md#ingestsource)
:   ​

`arguments` map of string to string
:   ​

`timestamp_metadata` [`ingest.v2.TimestampMetadata`](https://dev.nominal.io/api/grpc/ingest-service.md#ingest-v2-timestampmetadata)
:   ​

### CsvFormat

Set exactly one of the fields below.

one of `wide`, `long` required
:   ​

    **wide**

    `wide` [`WideFormat`](https://dev.nominal.io/api/grpc/ingest-service.md#wideformat)
    :   ​

    **long**

    `long` [`LongFormat`](https://dev.nominal.io/api/grpc/ingest-service.md#longformat)
    :   ​

### CsvIngestOptions

CSV ingest options.

data_row must be greater than header_row. units_row must not be header_row. units_row must be less than data_row, which defaults to header_row + 1.

`format` [`CsvFormat`](https://dev.nominal.io/api/grpc/ingest-service.md#csvformat) required
:   ​

`header_row` integer
:   One-based record number of the header, defaulting to 1; must be positive. Records are counted from 1, a multiline record counts as one, and blank lines are ignored. Greater than 0.

`data_row` integer
:   One-based record number of the first data record, defaulting to header_row + 1; must be greater than header_row. Records that fall between header_row and data_row are skipped. Greater than 0.

`units_row` integer
:   One-based record number holding each column’s unit; must not be header_row and must precede data_row, so reading a units row requires setting data_row past it. Cells match columns by position, so the record needs a placeholder cell for the timestamp column, and a column whose cell is empty gets no unit. Absent means no units row. FileIngestOptions.units overrides a unit read here, per channel. Greater than 0.

### CustomTimestamp

Absolute timestamps parsed with a user-supplied format string.

`format` string
:   The format string should be in the format of the `DateTimeFormatter` class in Java.

`default_year` integer
:   Default year is accepted as an optional field for cases like IRIG time format. Will be overridden by year in time format.

`default_day_of_year` integer
:   Default day of year is accepted as an optional field for cases like IRIG time format. Will be overridden by day of year in time format. From 1 to 366.

### DataflashIngestItem

One ArduPilot dataflash log conversion. Worker-mcap converts the log into an avro stream that is then ingested into the dataset.

`source` [`IngestSource`](https://dev.nominal.io/api/grpc/ingest-service.md#ingestsource) required
:   ​

### EpochTimestamp

Absolute timestamps expressed as an integer count of some time unit since the Unix epoch.

`time_unit` string
:   Unit in which epoch values are expressed (e.g. SECONDS, MILLISECONDS).

### FileIngestItem

Direct file ingest work.

`source` [`IngestSource`](https://dev.nominal.io/api/grpc/ingest-service.md#ingestsource) required
:   ​

`ingest` [`FileIngestOptions`](https://dev.nominal.io/api/grpc/ingest-service.md#fileingestoptions) required
:   ​

### FileIngestOptions

Direct file ingest mode.

Set exactly one of `csv`, `parquet`, `avro`.

`timestamp_metadata` [`ingest.v2.TimestampMetadata`](https://dev.nominal.io/api/grpc/ingest-service.md#ingest-v2-timestampmetadata) required
:   ​

`units` map of string to string
:   Unit per channel. Overrides the unit read from CsvIngestOptions.units_row for the channels named here; channels absent from this map keep their units-row unit.

`channel_prefix` string
:   ​

`channel_name_overrides` map of string to string
:   Renames individual columns to explicit channel names. Composes with channel_prefix: final name = channel_prefix + channel_name_overrides.getOrDefault(column, column). At most 1000 entries may be provided. Resolved channel names must be unique across columns; renaming a column onto another column’s name fails the ingest.

one of `csv`, `parquet`, `avro` required
:   ​

    **csv**

    `csv` [`CsvIngestOptions`](https://dev.nominal.io/api/grpc/ingest-service.md#csvingestoptions)
    :   ​

    **parquet**

    `parquet` [`ParquetIngestOptions`](https://dev.nominal.io/api/grpc/ingest-service.md#parquetingestoptions)
    :   ​

    **avro**

    `avro` [`AvroIngestOptions`](https://dev.nominal.io/api/grpc/ingest-service.md#avroingestoptions)
    :   ​

### FileStoreIngestSource

File Store source. The resolved file RID supplies provider-specific pinning and dereference metadata.

`file_revision_rid` string
:   Resolved File Store content target to ingest.

### GcsIngestSource

`path` string required
:   ​

### ingest.v2.TimestampMetadata

Timestamp metadata for ingest items that derive timestamps from an input column or field.

`column` string required
:   ​

`type` [`TimestampType`](https://dev.nominal.io/api/grpc/ingest-service.md#timestamptype) required
:   ​

### IngestItem

One unit of ingest work submitted under a single ingest job.

Set exactly one of `file`, `log`, `containerized`, `video`, `point_cloud`, `mcap`, `dataflash`.

`tags` map of string to string
:   Tags applied only to this ingest item. These override request-level tags on conflict.

one of 7 fields required
:   ​

    **file**

    `file` [`FileIngestItem`](https://dev.nominal.io/api/grpc/ingest-service.md#fileingestitem)
    :   ​

    **log**

    `log` [`LogIngestItem`](https://dev.nominal.io/api/grpc/ingest-service.md#logingestitem)
    :   ​

    **containerized**

    `containerized` [`ContainerizedIngestItem`](https://dev.nominal.io/api/grpc/ingest-service.md#containerizedingestitem)
    :   ​

    **video**

    `video` [`VideoIngestItem`](https://dev.nominal.io/api/grpc/ingest-service.md#videoingestitem)
    :   ​

    **point_cloud**

    `point_cloud` [`PointCloudIngestItem`](https://dev.nominal.io/api/grpc/ingest-service.md#pointcloudingestitem)
    :   ​

    **mcap**

    `mcap` [`McapIngestItem`](https://dev.nominal.io/api/grpc/ingest-service.md#mcapingestitem)
    :   ​

    **dataflash**

    `dataflash` [`DataflashIngestItem`](https://dev.nominal.io/api/grpc/ingest-service.md#dataflashingestitem)
    :   ​

### IngestSource

Source file location.

Set exactly one of the fields below.

one of `s3`, `gcs`, `presigned_file`, `file_store` required
:   ​

    **s3**

    `s3` [`S3IngestSource`](https://dev.nominal.io/api/grpc/ingest-service.md#s3ingestsource)
    :   ​

    **gcs**

    `gcs` [`GcsIngestSource`](https://dev.nominal.io/api/grpc/ingest-service.md#gcsingestsource)
    :   ​

    **presigned_file**

    `presigned_file` [`PresignedFileIngestSource`](https://dev.nominal.io/api/grpc/ingest-service.md#presignedfileingestsource)
    :   ​

    **file_store**

    `file_store` [`FileStoreIngestSource`](https://dev.nominal.io/api/grpc/ingest-service.md#filestoreingestsource)
    :   Initially supported for direct file ingest only.

### Iso8601Timestamp

Absolute timestamps serialized in ISO-8601 form. No additional parser configuration is needed.

A message with no fields.

### LogIngestItem

Single-channel log or journal ingest work.

`source` [`IngestSource`](https://dev.nominal.io/api/grpc/ingest-service.md#ingestsource) required
:   ​

`channel` string
:   ​

`timestamp_metadata` [`ingest.v2.TimestampMetadata`](https://dev.nominal.io/api/grpc/ingest-service.md#ingest-v2-timestampmetadata)
:   ​

`message_field` string
:   JSON field whose value becomes the log message. Defaults to the journald MESSAGE field.

### LongFormat

Long format: each row or record contains one channel, timestamp, and value sample.

`channel_column` string
:   Defaults to “channel”.

`value_column` string
:   Defaults to “value”.

`tags_column` string
:   Defaults to “tags”.

### McapChannelLocator

Locates the video’s channel within an mcap file. Topics are not guaranteed unique, so an mcap file with duplicate topics must be located by id.

Set exactly one of the fields below.

one of `topic`, `id` required
:   ​

    **topic**

    `topic` string
    :   ​

    **id**

    `id` integer
    :   ​

### McapChannelSelection

Selects mcap channels by topic name. All channels are ingested when neither list is set.

Set at most one of the fields below.

one of `include_topics`, `exclude_topics`
:   ​

    **include_topics**

    `include_topics` [`McapTopicNames`](https://dev.nominal.io/api/grpc/ingest-service.md#mcaptopicnames)
    :   ​

    **exclude_topics**

    `exclude_topics` [`McapTopicNames`](https://dev.nominal.io/api/grpc/ingest-service.md#mcaptopicnames)
    :   ​

### McapIngestItem

One mcap file conversion. Worker-mcap converts the file’s protobuf channels into an avro stream that is then ingested into the dataset.

`source` [`IngestSource`](https://dev.nominal.io/api/grpc/ingest-service.md#ingestsource) required
:   ​

`channels` [`McapChannelSelection`](https://dev.nominal.io/api/grpc/ingest-service.md#mcapchannelselection)
:   Which mcap channels to ingest, selected by topic name. All channels when unset.

`ignore_invalid_topics` boolean
:   Skip topics whose schemas cannot be decoded instead of failing the conversion.

### McapTimestampManifest

Frame timestamps are embedded in the mcap file that carries the video frames, rather than supplied as a sidecar manifest.

`channel_locator` [`McapChannelLocator`](https://dev.nominal.io/api/grpc/ingest-service.md#mcapchannellocator) required
:   ​

### McapTopicNames

`topics` array of string
:   ​

### NoTimestampManifest

`starting_timestamp` string (date-time) required
:   ​

`scale_parameter` [`ScaleParameter`](https://dev.nominal.io/api/grpc/ingest-service.md#scaleparameter)
:   ​

### ParquetFormat

Set exactly one of the fields below.

one of `wide`, `long` required
:   ​

    **wide**

    `wide` [`WideFormat`](https://dev.nominal.io/api/grpc/ingest-service.md#wideformat)
    :   ​

    **long**

    `long` [`LongFormat`](https://dev.nominal.io/api/grpc/ingest-service.md#longformat)
    :   ​

### ParquetIngestOptions

Parquet ingest options.

`format` [`ParquetFormat`](https://dev.nominal.io/api/grpc/ingest-service.md#parquetformat) required
:   ​

`is_archive` boolean
:   ​

### PointCloudIngestItem

Dataset-backed point cloud ingest work.

`source` [`IngestSource`](https://dev.nominal.io/api/grpc/ingest-service.md#ingestsource) required
:   ​

`ingest` [`PointCloudIngestOptions`](https://dev.nominal.io/api/grpc/ingest-service.md#pointcloudingestoptions) required
:   ​

### PointCloudIngestOptions

`sensor_metadata` [`PointCloudSensorMetadata`](https://dev.nominal.io/api/grpc/ingest-service.md#pointcloudsensormetadata)
:   ​

`channel` string required
:   ​

### PointCloudSensorMetadata

Placeholder for point-cloud sensor metadata until the spatial metadata proto is promoted.

`properties` map of string to string
:   ​

### PresignedFileIngestSource

`url` string required
:   ​

### RelativeTimestamp

Relative timestamp parser: interprets each row’s timestamp as an offset from a fixed origin.

`time_unit` string
:   Unit in which relative offsets are expressed. Aliases the classic REST TimeUnit alias.

`offset` string (date-time)
:   Starting timestamp to use when indexing the file.

### S3IngestSource

`path` string required
:   ​

### ScaleParameter

Set exactly one of the fields below.

one of `true_frame_rate`, `ending_timestamp`, `scale_factor` required
:   ​

    **true_frame_rate**

    `true_frame_rate` number
    :   ​

    **ending_timestamp**

    `ending_timestamp` string (date-time)
    :   ​

    **scale_factor**

    `scale_factor` number
    :   ​

### TimestampManifestFiles

Sidecar manifest files whose per-frame timestamps are concatenated in list order. Used when a video’s frame timestamps are recorded across several files.

`sources` array of [`IngestSource`](https://dev.nominal.io/api/grpc/ingest-service.md#ingestsource)
:   At least 1 item.

### TimestampType

Selects between relative-offset and absolute-timestamp parsing strategies.

Set exactly one of the fields below.

one of `relative`, `absolute` required
:   ​

    **relative**

    `relative` [`RelativeTimestamp`](https://dev.nominal.io/api/grpc/ingest-service.md#relativetimestamp)
    :   ​

    **absolute**

    `absolute` [`AbsoluteTimestamp`](https://dev.nominal.io/api/grpc/ingest-service.md#absolutetimestamp)
    :   ​

### VideoIngestItem

Dataset-file-backed video ingest work.

`source` [`IngestSource`](https://dev.nominal.io/api/grpc/ingest-service.md#ingestsource) required
:   ​

`ingest` [`VideoIngestOptions`](https://dev.nominal.io/api/grpc/ingest-service.md#videoingestoptions) required
:   ​

`additional_sources` array of [`IngestSource`](https://dev.nominal.io/api/grpc/ingest-service.md#ingestsource)
:   Videos appended in order after `source`, ingested as one continuous recording on the channel. Each becomes its own dataset file. The first file starts at the no-manifest starting timestamp and each later file starts where the previous one ends. Requires a no_manifest timestamp manifest and sources other than File Store; at most 99, so 100 files in all. At most 99 items.

### VideoIngestOptions

`timestamp_manifest` [`VideoTimestampManifest`](https://dev.nominal.io/api/grpc/ingest-service.md#videotimestampmanifest) required
:   ​

`channel` string required
:   ​

`overwrite_segments` boolean
:   ​

### VideoTimestampManifest

Set exactly one of the fields below.

one of `no_manifest`, `timestamp_manifest_files`, `mcap` required
:   ​

    **no_manifest**

    `no_manifest` [`NoTimestampManifest`](https://dev.nominal.io/api/grpc/ingest-service.md#notimestampmanifest)
    :   ​

    **timestamp_manifest_files**

    `timestamp_manifest_files` [`TimestampManifestFiles`](https://dev.nominal.io/api/grpc/ingest-service.md#timestampmanifestfiles)
    :   ​

    **mcap**

    `mcap` [`McapTimestampManifest`](https://dev.nominal.io/api/grpc/ingest-service.md#mcaptimestampmanifest)
    :   ​

### WideFormat

Wide format: each row or record contains one timestamp and many channels as columns.

`tag_columns` map of string to string
:   ​

`exclude_columns` array of string
:   ​

Page design based on [Scalar](https://scalar.com), MIT License.