AvroFileConsumer#
- pub struct AvroFileConsumer#
Implementations#
- impl AvroFileConsumer
- pub fn new(directory: impl Into<PathBuf>, file_prefix: Option<String>, dataset_rid: Option<ResourceIdentifier>) -> Result<Self>#
- pub fn new_with_full_path(file_path: impl Into<PathBuf>, overwrite: bool, dataset_rid: Option<ResourceIdentifier>) -> Result<Self>#
Opens
file_pathfor writing and wraps it in an avroWriter.If
overwriteis true and the path already exists, its prior contents are discarded. Truncation is required when reusing a path: the avro container format is single-header-and-blocks, so opening a longer existing file without truncating would leave leftover bytes from the previous run past the new content’s end and produce a corrupt reader stream.If
overwriteis false and the path already exists, anio::ErrorKind::AlreadyExistserror is returned and no file is touched. This is the safe choice when the caller does not want to silently destroy prior data.If
dataset_ridis provided, it is written to the avro file’s user metadata under thenominal.dataset_ridkey so downstream readers can identify the dataset the file belongs to.
- pub fn new_lazy(file_path: impl Into<PathBuf>, dataset_rid: Option<ResourceIdentifier>) -> Result<Self>#
Defers creating
file_pathuntil the first request is consumed, so a consumer that never receives a request leaves no file behind.An existing non-empty file at
file_pathis never modified: this returns anio::ErrorKind::AlreadyExistserror. The parent directory is created here so that an unusable location is reported up front.
Trait implementations#
- impl Debug for AvroFileConsumer
- impl WriteRequestConsumer for AvroFileConsumer
- fn consume(&self, request: &WriteRequestNominal) -> ConsumerResult<()>
- impl Drop for AvroFileConsumer
- fn drop(&mut self)
Defensive flush-on-drop. In normal operation, records reach disk via
append_series→apache_avro::Writer::extend, which flushes at the end of every call. Butapache_avro::Writeritself does not flush on drop, so any code path that bypassesextend(e.g. a directWriter::append, or a future writer call that forgets to flush) would silently lose buffered records when the consumer goes out of scope. This impl makes that failure mode impossible regardless of how the inner writer is driven.
Derived: Clone
Auto traits: Send, Sync, Unpin, !UnwindSafe, !RefUnwindSafe