Timestamps and tags#

Two choices decide whether an extractor’s data lines up and stays separable: how its timestamps are read, and which tags it applies. The examples use the Python package; the same levels exist in the manifest and the REST API.

Timestamps#

Nominal places every sample on an absolute timeline. Your output supplies a numeric or string timestamp per sample; timestamp metadata says how to read it.

Where metadata comes from, and what wins#

Three levels can specify it. Per output file, the pipeline resolves them in this order:

Precedence

Level

Set with

Scope

1 (highest)

Per output

ctx.add_tabular(..., timestamp_column=, timestamp_type=)

one file in one run

2

Ingest request

dataset.add_containerized(..., timestamp_column=, timestamp_type=)

every output of one run

3 (lowest)

Image default

@manifest_extractor(default_timestamp_column=, default_timestamp_type=), exported by .registration_kwargs()

every run of that image

Levels 2 and 3 are resolved before the container runs, and ingestion fails if both are absent. That is why registration requires a default: it guarantees the job always has metadata to fall back to. Level 1 is applied afterward, from the manifest your code writes, and overrides the resolved job-level value for that file only.

Use each level for what it describes:

  • The image default is the normal shape of this extractor’s output. Declare its column and type on the extractor decorator, then register using .registration_kwargs(). Single-file extractors accept the same timestamp options.

  • The ingest request is for facts about this upload that the image cannot know. Above all, this is where a relative t0 belongs: timestamp_type=ts.Relative("milliseconds", start=t0).

  • Per output is for a run whose files disagree on timestamp fields or units. A manifest extractor writing telemetry in microseconds and events in seconds needs this; nothing else does.

Per-output metadata takes numeric types only, ts.Epoch or ts.Relative, in units of seconds through nanoseconds. An output needing ISO 8601 or a custom string format must omit the per-output pair and inherit the job-level value, which accepts the full range. For an Avro output, inheriting is correct only when the job-level type is numeric too: Avro timestamps are integers, and a string format cannot read them.

# per output, in the manifest your extractor writes
ctx.add_tabular(part, timestamp_column="ts", timestamp_type="epoch_microseconds")
ctx.add_tabular(run, timestamp_column="elapsed", timestamp_type=ts.Relative("milliseconds", start=t0))

# avro takes the type alone: the schema fixes which field holds the timestamps
ctx.add_avro_stream(records, timestamp_type="epoch_microseconds")

Prefer relative timestamps#

Emit elapsed time from the start of the recording and supply t0 at ingest. Reach for absolute (ts.Epoch, ts.Iso8601, ts.Custom) only when you can guarantee every timestamp is already correct in absolute time.

Relative wins on two counts.

It is recoverable. With relative timestamps, absolute time is (elapsed in the file) + (t0 in metadata), so a wrong t0 is a metadata error: delete the file and re-ingest the same bytes with a corrected start. With absolute timestamps, the times are the data, so the only fix is to download the file, rewrite every value, and upload it again.

It is a smaller problem to get right. Relative asks one question per sample, how far into the recording it is, which the logger measures directly off its own clock. Absolute asks every sample to be correct in wall-clock time, which depends on GPS lock, NTP sync, and clock drift, any of which can leave you with jumps, rounding, or an offset nobody notices until someone compares two sources.

So: if the source records only elapsed time, emit relative. If the source records absolute time but you cannot vouch for it (unsynced machines, intermittent GPS, timestamps rounded to the second, occasional jumps), convert to elapsed-from-first-sample and emit relative with a best-estimate t0. Choosing absolute is a claim that no timestamp will ever need altering.

Do not register ts.Relative as an image’s default_timestamp_type. A default is set once and applies to every future ingest, so a fixed start gives every file the same t0. Register an absolute default for the fallback and pass Relative per ingest, where each file carries its own start.

Granularity#

Use the unit the data has. Declaring epoch_milliseconds for microsecond data does not lose precision; it misplaces every sample by a factor of 1000.

Tags#

Tags separate otherwise-identical channels. A channel is identified by name, so two stands both producing chamber_pressure blend into one series unless a tag distinguishes them. Blended data is harder to detect and harder to undo than a failed ingest.

Tags reach the data three ways, in increasing specificity:

  • The ingest request: add_containerized(..., tags={"vehicle": "n1234"}), applied to everything the run produces and readable in the container as ctx.additional_tags. Use it for facts about this upload that the file does not carry.

  • A tag column: ctx.add_tabular(path, tag_columns={"motor": "motor_id"}), read per row, when the distinguishing fact varies within a file.

  • Avro files and JSON Lines logs, which carry tags within the files themselves.

A good tag identifies a source, is stable for the life of the data, and has few distinct values: vehicle, stand, motor serial, run identifier. A bad tag is a measurement (that is a channel), a value derived from a timestamp (that is the timeline), or free text that varies by upload: Stand A, stand-a, and standA become three unrelated series. Nothing normalizes tag values, so normalize them in the extractor rather than relying on each caller.

channel_prefix and tags solve different problems. A prefix renames channels (engine/chamber_pressure); a tag leaves the name alone and adds a dimension you can filter and group by. Prefer a tag when comparing the same measurement across sources.