Stream files#
A stream can optionally write every point it streams to an Avro file.
This is different from a fallback file, which only stores data that couldn’t reach Nominal servers.
A stream can be configured to optionally transmit data to Nominal and either write all data to disk or store failed samples in a fallback file.
Write every point to a file#
This section covers the full redundancy case: stream to Nominal and also write all data to a file.
use nominal_streaming::prelude::*;
use nominal_streaming::stream::NominalDatasetStreamBuilder;
// Load the API key and base URL that API Keys Setup saved
let config = nominal::Config::load()?;
let profile = config.get_profile("default").ok_or("no profile named default")?;
let token = BearerToken::new(profile.token())?;
let dataset = ResourceIdentifier::new("<ri.catalog.instance.dataset.identifier>")?;
// Send to Nominal, and write every point to all-data-redundant.avro
let stream = NominalDatasetStreamBuilder::new()
.stream_to_core(token, dataset, tokio::runtime::Handle::current())
.with_options(NominalStreamOpts::default().with_base_api_url(profile.base_url()))
.stream_to_file("all-data-redundant.avro")
.build();
from pathlib import Path
from nominal.config import NominalConfig
from nominal_streaming import NominalDatasetStream, PyNominalStreamOpts
# Load the API key and base URL that API Keys Setup saved
profile = NominalConfig.from_yaml().get_profile("default")
# Send to Nominal, and write every point to all-data-redundant.avro
stream = (
NominalDatasetStream(auth_header=profile.token, opts=PyNominalStreamOpts(base_api_url=profile.base_url))
.with_core_consumer("<ri.catalog.instance.dataset.identifier>") #send to Nominal
.to_file(Path("all-data-redundant.avro"))
)
Once the stream is configured, all of the normal write methods work on the stream.
Disabling network transmission#
To record with no network, write only to a file. This requires no API key or dataset RID, but you must upload the file later.
use nominal_streaming::stream::NominalDatasetStreamBuilder;
// Write only to the file, with no API key or network
let offline = NominalDatasetStreamBuilder::new().stream_to_file("offline.avro").build();
import time
from pathlib import Path
from nominal_streaming import NominalDatasetStream
# Write only to the file, with no API key or network
with NominalDatasetStream().to_file(Path("offline.avro")) as offline:
offline.enqueue("engine.rpm", time.time_ns(), 3000.0)
Upload a file to Nominal#
Nominal ingests these files directly. Upload one with nomctl, installed when you configure your API keys or with a Nominal client:
nomctl --profile default ingest avro-stream --dataset <ri.catalog.instance.dataset.identifier> fallback_20260101_120000.avro
To ingest into a new dataset instead, replace --dataset and the RID with --name "<dataset name>".
With the nominal crate:
use nominal::core::AvroStreamIngest;
let client = nominal::NominalClient::from_profile("default")?;
let (job, _) = client
.ingest()
.upload_avro_stream("fallback_20260101_120000.avro", "<ri.catalog.instance.dataset.identifier>", AvroStreamIngest::new())
.await?;
client.ingest().wait_for_ingest_job(job.rid()).await?;
With the nominal package:
from nominal.core import NominalClient
client = NominalClient.from_profile("default")
dataset = client.get_dataset("<ri.catalog.instance.dataset.identifier>")
dataset.add_avro_stream("fallback_20260101_120000.avro").poll_until_ingestion_completed()
What’s in a file#
A stream file is an Avro object container file, compressed with Snappy. Each record in the file holds one batch of points for one channel and set of tags:
Field |
Type |
Holds |
|---|---|---|
|
string |
The channel name |
|
array of long |
Each point’s time, in nanoseconds since the Unix epoch |
|
array |
Each point’s value, as typed by the streaming library |
|
map of strings |
The points’ tags |
Warning
Avro has no unsigned 64-bit integers, so u64 values are stored by casting the value to i64.
This example uses apache-avro, added with cargo add apache-avro --features snappy.
To simplify handling of records, this example also uses serde, added with cargo add serde --features derive:
use std::collections::HashMap;
use apache_avro::Reader;
// A point's value: a number or string, or a record holding an array or a struct.
// Long comes before Double so that longs aren't read as floats.
#[derive(Debug, serde::Deserialize)]
#[serde(untagged)]
enum PointValue {
Long(i64),
Double(f64),
String(String),
DoubleArray { items: Vec<f64> },
StringArray { items: Vec<String> },
JsonStruct { json: String },
}
#[derive(serde::Deserialize)]
struct Batch {
channel: String,
tags: HashMap<String, String>,
timestamps: Vec<i64>,
values: Vec<PointValue>,
}
let file = std::fs::File::open("fallback_20260101_120000.avro")?;
for record in Reader::new(file)? {
let batch: Batch = apache_avro::from_value(&record?)?;
for (timestamp, value) in batch.timestamps.iter().zip(&batch.values) {
println!("{} {:?} {timestamp} {value:?}", batch.channel, batch.tags);
}
}
This example uses fastavro, installed with pip install "fastavro[snappy]":
from fastavro import reader
with open("fallback_20260101_120000.avro", "rb") as f:
for record in reader(f):
# Arrays come back as {"items": [...]}, and structs as {"json": "..."}
for timestamp, value in zip(record["timestamps"], record["values"]):
print(record["channel"], record["tags"], timestamp, value)