Upload files#
Upload a file into a dataset and ingest that file using Nominal making the file’s channels available.
To use this page, first make sure you can Connect to Nominal and obtain a client.
Upload a file#
Getting data from a file into Nominal takes two steps. First, an upload, which sends the file to Nominal, then an ingest job, which processes the uploaded file into time-series data Nominal understands. Each upload method does both, and returns the job so you can wait for it.
use nominal::core::{CsvIngest, DatasetCreate, Timestamp};
// Create a dataset, and upload the file into it
let (job, dataset_rid) = client
.ingest()
.upload_csv("bench.csv", DatasetCreate::new("Bench test"), CsvIngest::new(Timestamp::iso8601("timestamp")))
.await?;
// Wait for Nominal to ingest the file. This returns an error if the ingest fails.
client.ingest().wait_for_ingest_job(job.rid()).await?;
To upload into an existing dataset, pass its RID in place of the DatasetCreate.
Check an ingest job#
get_ingest_job checks a job once, without waiting. wait_for_ingest_job_with_interval waits like wait_for_ingest_job, at an interval you choose.
use std::time::Duration;
use nominal::core::IngestJobStatus;
// Check once, without waiting
let job = client.ingest().get_ingest_job(job.rid()).await?;
match job.status() {
IngestJobStatus::Completed => println!("ingested"),
IngestJobStatus::Submitted | IngestJobStatus::Queued | IngestJobStatus::InProgress => println!("still ingesting"),
other => eprintln!("not ingested: {other:?}"), // Failed, Cancelled, or a status this client doesn't know
}
// Or wait, checking every 10 seconds rather than every 2, such as for a large file
let job = client.ingest().wait_for_ingest_job_with_interval(job.rid(), Duration::from_secs(10)).await?;
File types#
Nominal ingests these file types itself, and each has its own upload method and options. Direct File Ingestion describes each format and the options it takes.
File type |
Method |
Options |
|---|---|---|
CSV |
||
Parquet, or a |
||
MCAP, with protobuf messages |
||
journald JSON logs ( |
||
Avro file from a stream ( |
||
ArduPilot DataFlash ( |
||
Video ( |
CSV and Parquet#
Each column except the timestamp column becomes a channel. The Timestamp parameter identifies the timestamp column in the file and the format of the time values.
Times written as |
|
|---|---|
ISO 8601 |
|
Numbers since the Unix epoch |
|
Numbers since the test started |
|
Another format |
|
Some legacy formats like IRIG drop information like year or day-of-year. When this happens, set the missing part with with_default_year or with_default_day_of_year.
Other options allow you to prefix channel names, tag rows, and skip columns:
use nominal::core::{CsvIngest, TimeUnit, Timestamp};
let ingest = CsvIngest::new(Timestamp::epoch("timestamp", TimeUnit::Seconds))
.channel_prefix("bench.") // voltage becomes bench.voltage
.tag_column("motor_id", "motor") // tag each row with motor_id, from the motor column
.additional_file_tag("test_stand", "stand-2") // tag every row in the file with test_stand=stand-2
.exclude_column("notes"); // skip the notes column
client.ingest().upload_csv("bench.csv", "<ri.catalog.instance.dataset.identifier>", ingest).await?;
ParquetIngest takes the same options. For an archive of Parquet files, add is_archive(true).
MCAP#
Choose an MCAP file’s topics with either include_topic or exclude_topic.
use nominal::core::McapIngest;
let ingest = McapIngest::new()
.exclude_topic("/diagnostics")
.ignore_invalid_topics(true); // skip topics Nominal can't read, rather than fail the ingest
client.ingest().upload_mcap("drive-12.mcap", "<ri.catalog.instance.dataset.identifier>", ingest).await?;
For the video in an MCAP file, see Video.
journald JSON logs#
A journald JSON file (.jsonl or .jsonl.gz) holds one log entry per line, with its time in __REALTIME_TIMESTAMP and its text in MESSAGE. All of a file’s log lines go to one channel, named logs unless you name another with channel.
use nominal::core::JournalJsonIngest;
// Put the log lines in a channel named system_logs
let ingest = JournalJsonIngest::new().channel("system_logs");
client.ingest().upload_journal_json("system.jsonl", "<ri.catalog.instance.dataset.identifier>", ingest).await?;
Avro streams#
An Avro stream file holds data that is already compatible with Nominal and the ingest process has no additional options.
Stream fallback files use this format.
use nominal::core::AvroStreamIngest;
// A stream's fallback file, into the dataset the stream writes to
client
.ingest()
.upload_avro_stream("fallback_20260101_120000.avro", "<ri.catalog.instance.dataset.identifier>", AvroStreamIngest::new())
.await?;
ArduPilot DataFlash#
Each field of each message in an ArduPilot DataFlash log (.bin) becomes a channel named <message>.<field>, such as IMU.GyrX. additional_file_tag allows the user to tag every point in the file with additional metadata.
use nominal::core::DataflashIngest;
// Tag every point with the vehicle that flew
let ingest = DataflashIngest::new().additional_file_tag("vehicle", "QC-456");
client.ingest().upload_ardupilot_dataflash("flight.bin", "<ri.catalog.instance.dataset.identifier>", ingest).await?;
Video#
A video goes into a video resource rather than a dataset. Pass a VideoCreate to create one, or an existing video’s RID.
use chrono::{DateTime, Utc};
use nominal::core::{VideoCreate, VideoIngest};
// A standard container format like mp4 only requires a start timestamp
let first_frame: DateTime<Utc> = "2026-01-01T12:00:00Z".parse()?;
client
.ingest()
.upload_video("nose-camera.mp4", VideoCreate::new("Nose camera"), VideoIngest::starting_at(first_frame))
.await?;
// The video on one topic of an MCAP file
client
.ingest()
.upload_video("drive-12.mcap", VideoCreate::new("Front camera"), VideoIngest::mcap_topic("/camera/front"))
.await?;
To associate a video with an asset, use add_video.
Upload options and progress#
The options of each upload method above, such as CsvIngest, take UploadOptions, which set how the file uploads and report its progress.
use nominal::core::{CsvIngest, Timestamp, UploadEvent, UploadOptions};
let upload_options = UploadOptions::new()
.chunk_size(16 * 1024 * 1024) // bytes in each part: 64 MiB by default, and at least 5 MiB
.max_concurrency(4) // parts uploaded at once: 8 by default
.max_retries(5) // retries for each part: 3 by default
.on_progress(|event| match event {
UploadEvent::Started { total_bytes, total_parts } => println!("{total_bytes} bytes in {total_parts} parts"),
UploadEvent::PartCompleted { part_number, .. } => println!("part {part_number} uploaded"),
UploadEvent::Completed { .. } => println!("upload complete"),
});
let ingest = CsvIngest::new(Timestamp::iso8601("timestamp")).upload_options(upload_options);
The progress callback runs inside the upload, so keep it quick to avoid slowing down the process.
Other file types#
For other file types, a containerized extractor performs the ingest step. The extractor must be registered in each workspace before it can be used.
From Rust, upload the file and start the extractor’s ingest through the REST API, as in the Ingest with an extractor example.