Stream files#

A stream can optionally write every point it streams to an Avro file.

This is different from a fallback file, which only stores data that couldn’t reach Nominal servers.

A stream can be configured to optionally transmit data to Nominal and either write all data to disk or store failed samples in a fallback file.

Write every point to a file#

This section covers the full redundancy case: stream to Nominal and also write all data to a file.

use nominal_streaming::prelude::*;
use nominal_streaming::stream::NominalDatasetStreamBuilder;

// Load the API key and base URL that API Keys Setup saved
let config = nominal::Config::load()?;
let profile = config.get_profile("default").ok_or("no profile named default")?;
let token = BearerToken::new(profile.token())?;
let dataset = ResourceIdentifier::new("<ri.catalog.instance.dataset.identifier>")?;

// Send to Nominal, and write every point to all-data-redundant.avro
let stream = NominalDatasetStreamBuilder::new()
    .stream_to_core(token, dataset, tokio::runtime::Handle::current())
    .with_options(NominalStreamOpts::default().with_base_api_url(profile.base_url()))
    .stream_to_file("all-data-redundant.avro")
    .build();
from pathlib import Path

from nominal.config import NominalConfig
from nominal_streaming import NominalDatasetStream, PyNominalStreamOpts

# Load the API key and base URL that API Keys Setup saved
profile = NominalConfig.from_yaml().get_profile("default")

# Send to Nominal, and write every point to all-data-redundant.avro
stream = (
    NominalDatasetStream(auth_header=profile.token, opts=PyNominalStreamOpts(base_api_url=profile.base_url))
    .with_core_consumer("<ri.catalog.instance.dataset.identifier>") #send to Nominal
    .to_file(Path("all-data-redundant.avro"))
)

Once the stream is configured, all of the normal write methods work on the stream.

Disabling network transmission#

To record with no network, write only to a file. This requires no API key or dataset RID, but you must upload the file later.

use nominal_streaming::stream::NominalDatasetStreamBuilder;

// Write only to the file, with no API key or network
let offline = NominalDatasetStreamBuilder::new().stream_to_file("offline.avro").build();
import time
from pathlib import Path

from nominal_streaming import NominalDatasetStream

# Write only to the file, with no API key or network
with NominalDatasetStream().to_file(Path("offline.avro")) as offline:
    offline.enqueue("engine.rpm", time.time_ns(), 3000.0)

Upload a file to Nominal#

Nominal ingests these files directly. Upload one with nomctl, installed when you configure your API keys or with a Nominal client:

nomctl --profile default ingest avro-stream --dataset <ri.catalog.instance.dataset.identifier> fallback_20260101_120000.avro

To ingest into a new dataset instead, replace --dataset and the RID with --name "<dataset name>".

With the nominal crate:

use nominal::core::AvroStreamIngest;

let client = nominal::NominalClient::from_profile("default")?;
let (job, _) = client
    .ingest()
    .upload_avro_stream("fallback_20260101_120000.avro", "<ri.catalog.instance.dataset.identifier>", AvroStreamIngest::new())
    .await?;
client.ingest().wait_for_ingest_job(job.rid()).await?;

With the nominal package:

from nominal.core import NominalClient

client = NominalClient.from_profile("default")
dataset = client.get_dataset("<ri.catalog.instance.dataset.identifier>")
dataset.add_avro_stream("fallback_20260101_120000.avro").poll_until_ingestion_completed()

What’s in a file#

A stream file is an Avro object container file, compressed with Snappy. Each record in the file holds one batch of points for one channel and set of tags:

Field

Type

Holds

channel

string

The channel name

timestamps

array of long

Each point’s time, in nanoseconds since the Unix epoch

values

array

Each point’s value, as typed by the streaming library

tags

map of strings

The points’ tags

Warning

Avro has no unsigned 64-bit integers, so u64 values are stored by casting the value to i64.

This example uses apache-avro, added with cargo add apache-avro --features snappy.

To simplify handling of records, this example also uses serde, added with cargo add serde --features derive:

use std::collections::HashMap;

use apache_avro::Reader;

// A point's value: a number or string, or a record holding an array or a struct.
// Long comes before Double so that longs aren't read as floats.
#[derive(Debug, serde::Deserialize)]
#[serde(untagged)]
enum PointValue {
    Long(i64),
    Double(f64),
    String(String),
    DoubleArray { items: Vec<f64> },
    StringArray { items: Vec<String> },
    JsonStruct { json: String },
}

#[derive(serde::Deserialize)]
struct Batch {
    channel: String,
    tags: HashMap<String, String>,
    timestamps: Vec<i64>,
    values: Vec<PointValue>,
}

let file = std::fs::File::open("fallback_20260101_120000.avro")?;
for record in Reader::new(file)? {
    let batch: Batch = apache_avro::from_value(&record?)?;
    for (timestamp, value) in batch.timestamps.iter().zip(&batch.values) {
        println!("{} {:?} {timestamp} {value:?}", batch.channel, batch.tags);
    }
}

This example uses fastavro, installed with pip install "fastavro[snappy]":

from fastavro import reader

with open("fallback_20260101_120000.avro", "rb") as f:
    for record in reader(f):
        # Arrays come back as {"items": [...]}, and structs as {"json": "..."}
        for timestamp, value in zip(record["timestamps"], record["values"]):
            print(record["channel"], record["tags"], timestamp, value)