AvroFileConsumer#

pub struct AvroFileConsumer#

Implementations#

impl AvroFileConsumer
pub fn new(directory: impl Into<PathBuf>, file_prefix: Option<String>, dataset_rid: Option<ResourceIdentifier>) -> Result<Self>#
pub fn new_with_full_path(file_path: impl Into<PathBuf>, overwrite: bool, dataset_rid: Option<ResourceIdentifier>) -> Result<Self>#

Opens file_path for writing and wraps it in an avro Writer.

If overwrite is true and the path already exists, its prior contents are discarded. Truncation is required when reusing a path: the avro container format is single-header-and-blocks, so opening a longer existing file without truncating would leave leftover bytes from the previous run past the new content’s end and produce a corrupt reader stream.

If overwrite is false and the path already exists, an io::ErrorKind::AlreadyExists error is returned and no file is touched. This is the safe choice when the caller does not want to silently destroy prior data.

If dataset_rid is provided, it is written to the avro file’s user metadata under the nominal.dataset_rid key so downstream readers can identify the dataset the file belongs to.

pub fn new_lazy(file_path: impl Into<PathBuf>, dataset_rid: Option<ResourceIdentifier>) -> Result<Self>#

Defers creating file_path until the first request is consumed, so a consumer that never receives a request leaves no file behind.

An existing non-empty file at file_path is never modified: this returns an io::ErrorKind::AlreadyExists error. The parent directory is created here so that an unusable location is reported up front.

pub fn path(&self) -> &Path#

The file this consumer writes to.

Trait implementations#

impl Debug for AvroFileConsumer
fn fmt(&self, f: &mut Formatter<'_>) -> Result
impl WriteRequestConsumer for AvroFileConsumer
fn consume(&self, request: &WriteRequestNominal) -> ConsumerResult<()>
impl Drop for AvroFileConsumer
fn drop(&mut self)

Defensive flush-on-drop. In normal operation, records reach disk via append_series → apache_avro::Writer::extend, which flushes at the end of every call. But apache_avro::Writer itself does not flush on drop, so any code path that bypasses extend (e.g. a direct Writer::append, or a future writer call that forgets to flush) would silently lose buffered records when the consumer goes out of scope. This impl makes that failure mode impossible regardless of how the inner writer is driven.

Derived: Clone

Auto traits: Send, Sync, Unpin, !UnwindSafe, !RefUnwindSafe