nominal_streaming#
Package Contents#
- class nominal_streaming.LogStreamStats#
- property accepted_records: int#
- property acknowledged_records: int#
- property backed_up_records: int#
- property failed_records: int#
- property requests: int#
Logical upload requests; transport retries do not increment this count.
- property buffered_bytes: int#
- property last_error: str | None#
- class nominal_streaming.PyNominalLogStreamOpts(*, max_request_bytes: int = 8388608, max_buffered_bytes: int = 67108864, max_points_per_batch: int = 10000, max_request_delay_secs: float = 0.25, num_upload_workers: int = 4, num_runtime_workers: int = 2, base_api_url: str = 'https://api.gov.nominal.io/api')#
Log stream options with read-only properties and fluent configuration methods.
- property max_request_bytes: int#
- with_max_request_bytes(value: int) Self#
- property max_buffered_bytes: int#
- with_max_buffered_bytes(value: int) Self#
- property max_points_per_batch: int#
- with_max_points_per_batch(value: int) Self#
- property max_request_delay_secs: float#
- with_max_request_delay_secs(value: float) Self#
- property num_upload_workers: int#
- with_num_upload_workers(value: int) Self#
- property base_api_url: str#
- with_api_base_url(value: str) Self#
- property num_runtime_workers: int#
- with_num_runtime_workers(value: int) Self#
- property user_agent: str#
- with_user_agent(name: str, version: str) Self#
- class nominal_streaming.PyNominalStreamOpts(*, max_points_per_batch: int = 250000, max_request_delay_secs: float = 0.1, max_buffered_requests: int = 4, num_upload_workers: int = 8, num_runtime_workers: int = 8, base_api_url: str = 'https://api.gov.nominal.io/api', track_metrics: bool = False)#
Configuration options for Nominal data streaming.
This class configures how data points are batched, buffered, and dispatched to the Nominal backend. It mirrors the Rust NominalStreamOpts structure, providing Pythonic accessors and fluent builder-style methods.
- property track_metrics: bool#
Whether runtime metric channels are enabled.
- with_track_metrics(enabled: bool) Self#
Enable or disable runtime metric channels.
- Parameters:
enabled
Whether to emit runtime metrics when this configuration is used.
- Returns:
The same configuration instance, updated for fluent chaining.
- property max_points_per_batch: int#
Maximum number of data points per record before dispatch.
- Returns:
The configured upper bound on points per record.
Example
>>> PyNominalStreamOpts().max_points_per_batch 250000
- property max_request_delay_secs: float#
Maximum delay before forcing a request flush.
- Returns:
The maximum time to wait before sending pending data, in seconds.
Example
>>> PyNominalStreamOpts().max_request_delay_secs > 0 True
- property max_buffered_requests: int#
Maximum number of requests that may be buffered concurrently.
- Returns:
The maximum number of buffered requests before backpressure is applied.
Example
>>> PyNominalStreamOpts().max_buffered_requests >= 0 True
- property num_upload_workers: int#
Number of concurrent dispatcher tasks used for network transmission.
- Returns:
The number of dispatcher tasks.
Example
>>> PyNominalStreamOpts().num_upload_workers >= 1 True
- property num_runtime_workers: int#
Number of runtime worker threads for internal processing.
- Returns:
The configured number of runtime workers.
Example
>>> PyNominalStreamOpts().num_runtime_workers 8
- property base_api_url: str#
Base URL for the Nominal API endpoint.
- Returns:
The fully-qualified base API URL used for streaming requests.
Example
>>> isinstance(PyNominalStreamOpts().base_api_url, str) True
- with_max_points_per_batch(n: int) Self#
Set the maximum number of points per record.
- Parameters:
n
Maximum number of data points to include in a single record.
- Returns:
The updated instance for fluent chaining.
Example
>>> opts = PyNominalStreamOpts().with_max_points_per_batch(1000)
- with_max_request_delay_secs(delay_secs: float) Self#
Set the maximum delay before forcing a request flush.
- Parameters:
delay_secs
Maximum time in seconds to wait before sending pending data.
- Returns:
The updated instance for fluent chaining.
Example
>>> opts = PyNominalStreamOpts().with_max_request_delay_secs(1.0)
- with_max_buffered_requests(n: int) Self#
Set the maximum number of requests that can be buffered concurrently.
- Parameters:
n
Maximum number of buffered requests.
- Returns:
The updated instance for fluent chaining.
Example
>>> opts = PyNominalStreamOpts().with_max_buffered_requests(200)
- with_num_upload_workers(n: int) Self#
Set the number of asynchronous dispatcher tasks.
- Parameters:
n
Number of dispatcher tasks responsible for request transmission.
- Returns:
The updated instance for fluent chaining.
Example
>>> opts = PyNominalStreamOpts().with_num_upload_workers(8)
- with_num_runtime_workers(n: int) Self#
Set the number of runtime worker threads.
- Parameters:
n
Number of background worker threads used for internal processing.
- Returns:
The updated instance for fluent chaining.
Example
>>> opts = PyNominalStreamOpts().with_num_runtime_workers(16)
- with_api_base_url(url: str) Self#
Set the base URL for the Nominal API.
- Parameters:
url
Fully-qualified base API URL for streaming requests.
- Returns:
The updated instance for fluent chaining.
Example
>>> opts = PyNominalStreamOpts().with_api_base_url("https://staging.nominal.io")
- property user_agent: str#
The
name/versionsent first in theUser-Agentheader.- Returns:
The agent set by
with_user_agent, ornominal-streaming/<version>if none is set.
- with_user_agent(name: str, version: str) Self#
Set the program name and version sent first in the
User-Agentheader.Nominal’s usage reporting groups requests by this agent, and
nominal-streaming/<version>follows it.- Parameters:
name
Program name, such as
"nominal-python". Starts with an ASCII letter and contains only ASCII letters, digits and hyphens.version
Program version, such as
"1.2.0". Dot-separated numbers, optionally followed by-rcNand agit describesuffix such as-5-gabc123.
- Returns:
The updated instance for fluent chaining.
- Raises:
ValueError – If the name or version is not in the accepted format.
Example
>>> opts = PyNominalStreamOpts().with_user_agent("nominal-python", "1.2.0") >>> opts.user_agent 'nominal-python/1.2.0'
- class nominal_streaming.NominalDatasetStream(auth_header: str | None = None, opts: nominal_streaming._nominal_streaming.PyNominalStreamOpts | None = None)#
Top-level python wrapper for the Rust streaming client to Nominal.
- classmethod create(auth_header: str, base_api_url: str, max_points_per_batch: int = 250000, max_request_delay_secs: float = 0.1, max_buffered_requests: int = 4, num_upload_workers: int = 8, num_runtime_workers: int = 8, track_metrics: bool = False) Self#
Factory constructor to build a NominalDatasetStream using optional overrides for configuration options
- Parameters:
auth_header
API Key or Personal Access Token for accessing the Nominal API
base_api_url
Base API URL for hitting the Nominal API with.
max_points_per_batch
Overrides the default number of points that may be sent in a single batch
max_request_delay_secs
Overrides the default maximum buffering time for data between flushes. NOTE: if the amount of data being streamed is greater than available bandwidth, data may be buffered longer than the configured duration.
max_buffered_requests
Overrides the default number of requests that may be buffered between encoding threads and upload threads. Increasing this may prevent blocking threads in situations with spotty internet, but increase teardown time (e.g. when pressing ctrl + c)
num_upload_workers
Overrides the default number of upload worker threads NOTE: must be set as low as the number of runtime workers.
track_metrics
Emit runtime metric channels; disabled by default.
num_runtime_workers
Overrides the default number of runtime worker threads NOTE: must be set as high as the number of upload workers.
- enable_logging(log_directive: str = 'debug') Self#
Enable logging with the given verbosity level
- Parameters:
log_directive
Log verbosity level to expose from Rust code. Defaults to verbose debug logging. See the following for valid values: https://docs.rs/env_logger/latest/env_logger/#enabling-logging
- with_core_consumer(dataset_rid: str) Self#
Enables streaming to a Dataset in Core
- Parameters:
dataset_rid
RID of the Dataset in Nominal to stream to
- Raises:
RuntimeError – If no auth_header was provided to the constructor and the NOMINAL_TOKEN environment variable is not set.
- to_file(path: pathlib.Path) Self#
Target streaming towards a local .avro file
The written file will contain snappy-compressed avro data. This can be read as follows:
from fastavro import reader with open("test.avro", "rb") as f: for record in reader(f): channel_name = record["channel"] tags = record["tags"] timestamps = record["timestamps"] values = record["values"]
- with_file_fallback(path: pathlib.Path) Self#
Setup file fallback for streaming to core
The written file will contain snappy-compressed avro data for any batches of data that were unable to make it to the backend successfully. The file is created when a batch first fails, so a stream that delivers everything leaves no file behind. The stream’s start time (UTC) is appended to the file name, so each run writes to its own file: fallback.avro becomes fallback_20260101_120000.avro. The file can be read as follows:
from fastavro import reader with open("test.avro", "rb") as f: for record in reader(f): channel_name = record["channel"] tags = record["tags"] timestamps = record["timestamps"] values = record["values"]
- open() Self#
Create the stream as a context manager.
- NOTE: installs a sigint handler to enable more graceful shutdown.
This is restored upon exit. The handler is only installed when running in the main thread; in worker threads, the stream still functions but Ctrl+C will not trigger graceful cancellation.
- close(wait: bool = True) None#
Exit the stream and close out any used system resources.
NOTE: uninstalls the installed sigint handler and restores any pre-existing sigint handlers
- enqueue(channel_name: str, timestamp: TimestampLike, value: DataType, tags: Mapping[str, str] | None = None) None#
Write a single value to the stream
- Parameters:
channel_name
Name of the channel to upload data for.
timestamp
Absolute UTC timestamp of the data being uploaded.
value
Value to write to the specified channel.
tags
Key-value tags associated with the data being uploaded.
- enqueue_batch(channel_name: str, timestamps: Sequence[TimestampLike] | numpy.typing.NDArray[numpy.integer[Any]], values: Sequence[DataType] | numpy.typing.NDArray[numpy.integer[Any] | numpy.floating[Any] | numpy.bool_ | numpy.str_], tags: Mapping[str, str] | None = None) None#
Add a sequence of messages to the queue to upload to Nominal.
Messages are added one-by-one (with timestamp normalization) and flushed based on the batch conditions.
- NOTE: assumes that all values have the same type as the first value in the batch–
ensure that any provided value arrays are homogenously typed
Lists, tuples and numpy arrays are accepted. Supported numeric arrays are copied directly into Rust-owned memory; pass arrays directly rather than using .tolist().
Numpy arrays whose elements would be silently reinterpreted are rejected rather than written: datetime64/timedelta64 values (which would become raw epoch counts) and masked arrays with masked elements (which would become NaN). Convert explicitly if that is what you want.
- Parameters:
channel_name
Name of the channel to upload data for.
timestamps
Absolute UTC timestamps of the data being uploaded.
values
Values to write to the specified channel.
tags
Key-value tags associated with the data being uploaded.
- enqueue_from_dict(timestamp: TimestampLike, channel_values: Mapping[str, DataType], tags: Mapping[str, str] | None = None) None#
Write multiple channel values at a given timestamp using a flattened dictionary.
Each key in the dictionary is treated as a channel name and the corresponding value is enqueued with the given timestamp.
- Parameters:
timestamp
The shared absolute UTC timestamp to use for all items to enqueue.
channel_values
A dictionary mapping channel names to their respective values.
tags
Key-value tags associated with the data being uploaded.
- enqueue_struct(channel_name: str, timestamp: TimestampLike, value: dict[str, Any], tags: Mapping[str, str] | None = None) None#
Write a single struct value (JSON-encoded inside Rust) to the stream.
- Parameters:
channel_name
Name of the channel to upload data for.
timestamp
Absolute UTC timestamp of the data being uploaded.
value
Mapping to serialize as a JSON struct. Nested values must be JSON-native (int, float, str, bool, None, list, dict). Non-native types (datetime, numpy scalars, etc.) must be converted by the caller.
tags
Key-value tags associated with the data being uploaded.
- Raises:
TypeError – If value contains a non-JSON-native element.
- enqueue_float_array(channel_name: str, timestamp: TimestampLike, value: list[float], tags: Mapping[str, str] | None = None) None#
Write a single array-of-doubles value to the stream.
- Parameters:
channel_name
Name of the channel to upload data for.
timestamp
Absolute UTC timestamp of the data being uploaded.
value
List of doubles forming the array value at this timestamp. Integer elements are coerced to float; pass an explicit float sequence if implicit int-to-float promotion is undesired.
tags
Key-value tags associated with the data being uploaded.
- enqueue_string_array(channel_name: str, timestamp: TimestampLike, value: list[str], tags: Mapping[str, str] | None = None) None#
Write a single array-of-strings value to the stream.
- Parameters:
channel_name
Name of the channel to upload data for.
timestamp
Absolute UTC timestamp of the data being uploaded.
value
List of strings forming the array value at this timestamp.
tags
Key-value tags associated with the data being uploaded.
- class nominal_streaming.NominalLogStream(auth_header: str | None = None, opts: nominal_streaming._nominal_streaming.PyNominalLogStreamOpts | None = None)#
Log records are acknowledged by Core or preserved in a local journal.
Integer timestamps are signed Unix nanoseconds. Datetimes must be timezone aware. Configure a target before opening. Instances cannot be reopened after close. No process-wide signal handlers are installed.
- classmethod create(auth_header: str, base_api_url: str, max_points_per_batch: int = 10000, max_request_delay_secs: float = 0.25, num_upload_workers: int = 4, num_runtime_workers: int = 2, *, max_request_bytes: int = 8 * 1024 * 1024, max_buffered_bytes: int = 64 * 1024 * 1024) Self#
Create a stream with connection, batching and runtime options.
- Parameters:
auth_header
API key or access token for Nominal.
base_api_url
Base URL of the Nominal API.
max_points_per_batch
Maximum number of log messages per request.
max_request_delay_secs
Maximum buffering delay when capacity is available.
num_upload_workers
Number of upload workers.
num_runtime_workers
Number of asynchronous I/O runtime workers.
max_request_bytes
Maximum uncompressed protobuf request size.
max_buffered_bytes
Encoded size limit for all records held in memory.
- enable_logging(log_directive: str = 'debug') Self#
Enable Rust stream logging when the stream opens.
- Parameters:
log_directive
Tracing filter directive, such as “info” or “nominal_streaming=debug”.
- with_options(opts: nominal_streaming._nominal_streaming.PyNominalLogStreamOpts) Self#
Set stream options before opening.
- Parameters:
opts
Batching, buffering and runtime configuration.
- with_core_consumer(dataset_rid: str) Self#
- with_file_fallback(path: str | pathlib.Path) Self#
- to_file(path: str | pathlib.Path) Self#
- open() Self#
- close(wait: bool = True) nominal_streaming._nominal_streaming.LogStreamStats | None#
Stop writes and drain. With wait=False, drain in the background.
A later close(wait=True) waits for that drain and reports delivery errors.
- stop_accepting_writes() None#
- flush() nominal_streaming._nominal_streaming.LogStreamStats#
- stats() nominal_streaming._nominal_streaming.LogStreamStats#
- enqueue(channel_name: str, timestamp: TimestampLike, value: str, tags: Mapping[str, str] | None = None, *, args: Mapping[str, str] | None = None) None#
- enqueue_from_dict(timestamp: TimestampLike, channel_values: Mapping[str, str], tags: Mapping[str, str] | None = None, *, args: Mapping[str, str] | None = None) None#