Stream configuration#
Stream options set where points go, how they’re batched, and when writes wait for the network. A stream logs the requests that fail, and sends what it has buffered when you close it.
Stream options#
Options go in a NominalStreamOpts in Rust or a PyNominalStreamOpts in Python, which you pass to the stream when you create it:
use std::time::Duration;
use nominal_streaming::prelude::*;
use nominal_streaming::stream::NominalDatasetStreamBuilder;
// Load the API key and base URL that API Keys Setup saved
let config = nominal::Config::load()?;
let profile = config.get_profile("default").ok_or("no profile named default")?;
let token = BearerToken::new(profile.token())?;
let dataset = ResourceIdentifier::new("<ri.catalog.instance.dataset.identifier>")?;
let opts = NominalStreamOpts::default()
.with_base_api_url(profile.base_url())
.with_max_points_per_record(100_000)
.with_max_request_delay(Duration::from_millis(250));
let stream = NominalDatasetStreamBuilder::new()
.stream_to_core(token, dataset, tokio::runtime::Handle::current())
.with_options(opts)
.with_file_fallback("fallback.avro")
.build();
from nominal.config import NominalConfig
from nominal_streaming import NominalDatasetStream, PyNominalStreamOpts
# Load the API key and base URL that API Keys Setup saved
profile = NominalConfig.from_yaml().get_profile("default")
opts = PyNominalStreamOpts(
base_api_url=profile.base_url,
max_points_per_batch=100_000,
max_request_delay_secs=0.25,
)
stream = NominalDatasetStream(auth_header=profile.token, opts=opts)
# ALTERNATE:
# Pass the options straight to create, with no PyNominalStreamOpts
stream = NominalDatasetStream.create(
profile.token,
profile.base_url,
max_points_per_batch=100_000,
max_request_delay_secs=0.25,
)
Available Options#
Rust ( |
Python ( |
Default |
Sets |
|---|---|---|---|
|
The base URL from API Keys Setup |
||
250,000 |
The most points in one request |
||
100 ms |
How long points wait for a fuller request before it’s sent anyway |
||
4 |
How many requests can wait to be sent before writes wait too |
||
8 |
How many requests are sent at once |
||
8 |
Threads the stream runs on; at least |
||
off |
Metric channels about the stream’s latency |
Batching and backpressure#
The request size (with_max_points_per_record, or max_points_per_batch in Python) and the delay (with_max_request_delay, or max_request_delay_secs) set how points are batched into requests.
The option with_max_buffered_requests (max_buffered_requests in Python) sets how many requests can wait to be sent before backpressure makes your writes wait too.
Metrics#
With metrics on, the stream adds these channels to the dataset, in the requests it already sends. Each point’s timestamp is when its request finished, and its value is in seconds.
Channel |
Value |
|---|---|
|
How long the request took, including retries |
|
Time from the request’s oldest point to when it was sent |
|
Time from the request’s newest point to when it was sent |
|
Time from the request’s oldest point to when it finished |
|
Time from the request’s newest point to when it finished |
Reporting and handling errors#
A stream retries failed requests. A request that still fails goes to the fallback file, if configured.
Warning
Your writes don’t return streaming errors. Turn on logging to see it.
let stream = NominalDatasetStreamBuilder::new()
.stream_to_core(token, dataset, tokio::runtime::Handle::current())
.with_file_fallback("fallback.avro")
.enable_logging_with_directive("warn") // or enable_logging() for debug and up
.try_build()?; // build() panics instead if the stream can't start
To handle failures in your own code, add a NominalStreamListener with add_listener.
stream = (
NominalDatasetStream(auth_header=profile.token, opts=PyNominalStreamOpts(base_api_url=profile.base_url))
.enable_logging("warn") # or "debug" (the default), "info" or "error"
.with_core_consumer("<ri.catalog.instance.dataset.identifier>")
)
Log messages print to standard output. A warning that a request failed means its points went to the fallback file, so upload it when the network is back. An error that a request failed means its points were lost.
Close the stream#
Closing a stream sends what it has buffered.