Native extractor service#
Workspace-facing surface for enabling and converging native catalog extractors.
OpenAPI file: nominal.ingest.v2.json
Endpoints#
List native extractors#
GET /extractors/v2/native
Lists every catalog entry the deployment carries with this workspace’s installation state.
Query parameters
workspaceRidstring required
pageSizeinteger requiredFrom 1 to 1,000.
nextPageTokenstring
Response
extractorsarray ofNativeExtractor
Fields of
NativeExtractorcatalogIdstringStable catalog identifier, e.g. “ardupilot-dataflash”.
availableVersionsarray of stringVersions the deployment’s catalog currently carries, ascending.
installationContainerizedExtractorThe workspace-local extractor materialized for this entry, set when enabled. Ingest requests target it exactly like a customer-authored extractor, and the catalog version it is installed at is its active image’s tag.
Fields of
ContainerizedExtractorridstringUnique resource identifier for this extractor.
workspaceRidstringWorkspace that owns this extractor.
namestringUser-supplied name. Need not be unique within a workspace.
descriptionstringFree-form description shown in the UI.
createdAtstring (date-time)Timestamp at which this extractor was first registered.
isArchivedbooleanTrue once the extractor has been archived. Archived extractors are hidden from default search results and reject new ingests; flip back to false on UpdateContainerizedExtractor to restore.
activeContainerImageContainerImageThe container image currently selected to run this extractor. Unset until the first image for this extractor reaches READY status; ingest requests that arrive before then will fail. The schema fields (inputs, parameters, output_file_format, default_timestamp_metadata) live on this nested image.
Fields of
ContainerImageridstring
tagstringUser-supplied tag, unique within (workspace, extractor).
sizeBytesstring (int64)Size of the stored image in bytes once the image reaches READY.
statusContainerImageStatusCurrent lifecycle status (PENDING while pushing, READY after a successful push, FAILED otherwise).
Values of
ContainerImageStatusCONTAINER_IMAGE_STATUS_UNSPECIFIED
CONTAINER_IMAGE_STATUS_PENDING
CONTAINER_IMAGE_STATUS_READY
CONTAINER_IMAGE_STATUS_FAILED
createdAtstring (date-time)When the image record was created.
extractorRidstringThe containerized extractor this image is built for.
inputsarray ofFileExtractionInputThe input files this extractor requires, mapped to environment variables that store the path to each file.
Fields of
FileExtractionInputenvironmentVariablestringThe environment variable that stores the path to the input file.
namestringName of the input file which users will be prompted with.
descriptionstringDescription of the input file which users will be prompted with.
fileFiltersarray ofFileFilterOptionally filter files for file selection.
Fields of
FileFiltersuffixFileSuffix
Fields of
FileSuffixsuffixstring
requiredbooleanWhether the input file is required for the extractor to run.
parametersarray ofFileExtractionParameterParameters the extractor accepts.
Fields of
FileExtractionParameterenvironmentVariablestringThe environment variable that stores the argument.
namestringName of the parameter which users will be prompted with.
descriptionstringDescription of the parameter which users will be prompted with.
requiredbooleanWhether the parameter is required for the extractor to run.
fileOutputFormatFileOutputFormatFormat of the output file produced by the extractor.
Values of
FileOutputFormatFILE_OUTPUT_FORMAT_UNSPECIFIED
FILE_OUTPUT_FORMAT_PARQUET
FILE_OUTPUT_FORMAT_CSV
FILE_OUTPUT_FORMAT_PARQUET_TAR
FILE_OUTPUT_FORMAT_AVRO_STREAM
FILE_OUTPUT_FORMAT_JSON_L
FILE_OUTPUT_FORMAT_MANIFEST
defaultTimestampMetadataregistry.v2.TimestampMetadataMetadata about the intermediate parquet this extractor will produce. If not set, timestamp metadata must be provided at ingest time.
Fields of
registry.v2.TimestampMetadataseriesNamestringName of the series within the output whose values are used as the row timestamp.
timestampTypeTimestampTypeHow the values in
series_nameshould be interpreted as timestamps.Fields of
TimestampType- one of
relative,absolute
relativeRelativeTimestamp
Fields of
RelativeTimestamptimeUnitstringUnit in which relative offsets are expressed. Aliases the classic REST TimeUnit alias.
offsetstring (date-time)Starting timestamp to use when indexing the file.
absoluteAbsoluteTimestamp
Fields of
AbsoluteTimestamp- one of
iso8601,epochOfTimeUnit,customFormat
iso8601Iso8601Timestamp
epochOfTimeUnitEpochTimestamp
Fields of
EpochTimestamptimeUnitstring
Request
curl "https://api.gov.nominal.io/api/extractors/v2/native?workspaceRid=<workspace rid>&pageSize=1" \ -H "Authorization: Bearer <api key>"
import requests response = requests.get( "https://api.gov.nominal.io/api/extractors/v2/native?workspaceRid=<workspace rid>&pageSize=1", headers={"Authorization": "Bearer <api key>"}, ) response.raise_for_status() print(response.json())
import httpx response = httpx.get( "https://api.gov.nominal.io/api/extractors/v2/native?workspaceRid=<workspace rid>&pageSize=1", headers={"Authorization": "Bearer <api key>"}, ) response.raise_for_status() print(response.json())
import asyncio import aiohttp async def main() -> None: async with aiohttp.ClientSession() as session: async with session.get( "https://api.gov.nominal.io/api/extractors/v2/native?workspaceRid=<workspace rid>&pageSize=1", headers={"Authorization": "Bearer <api key>"}, ) as response: response.raise_for_status() print(await response.json()) asyncio.run(main())
Response:
200{ "extractors": [ { "catalogId": "<catalog id>", "availableVersions": [ "<available versions>" ], "installation": { "rid": "<rid>", "workspaceRid": "<workspace rid>", "name": "<name>", "description": "<description>", "createdAt": "2026-01-01T00:00:00Z", "isArchived": true, "activeContainerImage": { "rid": "<rid>", "tag": "<tag>", "sizeBytes": "1", "status": "CONTAINER_IMAGE_STATUS_PENDING", "createdAt": "2026-01-01T00:00:00Z", "extractorRid": "<extractor rid>", "inputs": [ { "environmentVariable": "<environment variable>", "name": "<name>", "description": "<description>", "fileFilters": [ { "suffix": { "suffix": "<suffix>" } } ], "required": true } ], "parameters": [ { "environmentVariable": "<environment variable>", "name": "<name>", "description": "<description>", "required": true } ], "fileOutputFormat": "FILE_OUTPUT_FORMAT_PARQUET", "defaultTimestampMetadata": { "seriesName": "<series name>", "timestampType": { "relative": { "timeUnit": "<time unit>", "offset": "2026-01-01T00:00:00Z" } } }, "exitCodeMappings": [ { "exitCode": 1, "code": "<code>", "message": "<message>", "retryable": true } ], "resources": { "cpuCores": 1, "memoryGib": 1, "diskGib": 1 } }, "origin": "EXTRACTOR_ORIGIN_CUSTOMER", "catalogId": "<catalog id>", "properties": { "<key>": "<value>" }, "labels": [ "<labels>" ] }, "desiredVersion": "<desired version>", "displayName": "<display name>", "description": "<description>" } ], "nextPageToken": "<next page token>" }
Disable native extractor#
POST
/extractors/v2/native/{catalogId}/disableArchives the workspace’s extractor, which is the installation; re-enabling unarchives it.
Path parameters
catalogIdstring required
Body
workspaceRidstring required
Response
An empty JSON object,
{}.Request
curl -X POST "https://api.gov.nominal.io/api/extractors/v2/native/<catalog id>/disable" \ -H "Authorization: Bearer <api key>" \ -H "Content-Type: application/json" \ --data '{ "workspaceRid": "<workspace rid>" }'
import requests response = requests.post( "https://api.gov.nominal.io/api/extractors/v2/native/<catalog id>/disable", headers={"Authorization": "Bearer <api key>"}, json={"workspaceRid": "<workspace rid>"}, ) response.raise_for_status() print(response.json())
import httpx response = httpx.post( "https://api.gov.nominal.io/api/extractors/v2/native/<catalog id>/disable", headers={"Authorization": "Bearer <api key>"}, json={"workspaceRid": "<workspace rid>"}, ) response.raise_for_status() print(response.json())
import asyncio import aiohttp async def main() -> None: async with aiohttp.ClientSession() as session: async with session.post( "https://api.gov.nominal.io/api/extractors/v2/native/<catalog id>/disable", headers={"Authorization": "Bearer <api key>"}, json={"workspaceRid": "<workspace rid>"}, ) as response: response.raise_for_status() print(await response.json()) asyncio.run(main())
Response:
200{}Enable native extractor#
POST
/extractors/v2/native/{catalogId}/enableEnables a catalog entry in a workspace and converges it to the deployment’s current desired version, materializing the workspace-local extractor and importing its container image. Idempotent: calling it on an already-enabled entry re-converges to the current desired version (this is also how a stale installation picks up a newly published version).
Path parameters
catalogIdstring required
Body
workspaceRidstring required
Response
extractorNativeExtractor
Fields of
NativeExtractorcatalogIdstringStable catalog identifier, e.g. “ardupilot-dataflash”.
availableVersionsarray of stringVersions the deployment’s catalog currently carries, ascending.
installationContainerizedExtractorThe workspace-local extractor materialized for this entry, set when enabled. Ingest requests target it exactly like a customer-authored extractor, and the catalog version it is installed at is its active image’s tag.
Fields of
ContainerizedExtractorridstringUnique resource identifier for this extractor.
workspaceRidstringWorkspace that owns this extractor.
namestringUser-supplied name. Need not be unique within a workspace.
descriptionstringFree-form description shown in the UI.
createdAtstring (date-time)Timestamp at which this extractor was first registered.
isArchivedbooleanTrue once the extractor has been archived. Archived extractors are hidden from default search results and reject new ingests; flip back to false on UpdateContainerizedExtractor to restore.
activeContainerImageContainerImageThe container image currently selected to run this extractor. Unset until the first image for this extractor reaches READY status; ingest requests that arrive before then will fail. The schema fields (inputs, parameters, output_file_format, default_timestamp_metadata) live on this nested image.
Fields of
ContainerImageridstring
tagstringUser-supplied tag, unique within (workspace, extractor).
sizeBytesstring (int64)Size of the stored image in bytes once the image reaches READY.
statusContainerImageStatusCurrent lifecycle status (PENDING while pushing, READY after a successful push, FAILED otherwise).
Values of
ContainerImageStatusCONTAINER_IMAGE_STATUS_UNSPECIFIED
CONTAINER_IMAGE_STATUS_PENDING
CONTAINER_IMAGE_STATUS_READY
CONTAINER_IMAGE_STATUS_FAILED
createdAtstring (date-time)When the image record was created.
extractorRidstringThe containerized extractor this image is built for.
inputsarray ofFileExtractionInputThe input files this extractor requires, mapped to environment variables that store the path to each file.
Fields of
FileExtractionInputenvironmentVariablestringThe environment variable that stores the path to the input file.
namestringName of the input file which users will be prompted with.
descriptionstringDescription of the input file which users will be prompted with.
fileFiltersarray ofFileFilterOptionally filter files for file selection.
Fields of
FileFiltersuffixFileSuffix
Fields of
FileSuffixsuffixstring
requiredbooleanWhether the input file is required for the extractor to run.
parametersarray ofFileExtractionParameterParameters the extractor accepts.
Fields of
FileExtractionParameterenvironmentVariablestringThe environment variable that stores the argument.
namestringName of the parameter which users will be prompted with.
descriptionstringDescription of the parameter which users will be prompted with.
requiredbooleanWhether the parameter is required for the extractor to run.
fileOutputFormatFileOutputFormatFormat of the output file produced by the extractor.
Values of
FileOutputFormatFILE_OUTPUT_FORMAT_UNSPECIFIED
FILE_OUTPUT_FORMAT_PARQUET
FILE_OUTPUT_FORMAT_CSV
FILE_OUTPUT_FORMAT_PARQUET_TAR
FILE_OUTPUT_FORMAT_AVRO_STREAM
FILE_OUTPUT_FORMAT_JSON_L
FILE_OUTPUT_FORMAT_MANIFEST
defaultTimestampMetadataregistry.v2.TimestampMetadataMetadata about the intermediate parquet this extractor will produce. If not set, timestamp metadata must be provided at ingest time.
Fields of
registry.v2.TimestampMetadataseriesNamestringName of the series within the output whose values are used as the row timestamp.
timestampTypeTimestampTypeHow the values in
series_nameshould be interpreted as timestamps.Fields of
TimestampType- one of
relative,absolute
relativeRelativeTimestamp
Fields of
RelativeTimestamptimeUnitstringUnit in which relative offsets are expressed. Aliases the classic REST TimeUnit alias.
offsetstring (date-time)Starting timestamp to use when indexing the file.
absoluteAbsoluteTimestamp
Fields of
AbsoluteTimestamp- one of
iso8601,epochOfTimeUnit,customFormat
iso8601Iso8601Timestamp
epochOfTimeUnitEpochTimestamp
Fields of
EpochTimestamptimeUnitstring
Request
curl -X POST "https://api.gov.nominal.io/api/extractors/v2/native/<catalog id>/enable" \ -H "Authorization: Bearer <api key>" \ -H "Content-Type: application/json" \ --data '{ "workspaceRid": "<workspace rid>" }'
import requests response = requests.post( "https://api.gov.nominal.io/api/extractors/v2/native/<catalog id>/enable", headers={"Authorization": "Bearer <api key>"}, json={"workspaceRid": "<workspace rid>"}, ) response.raise_for_status() print(response.json())
import httpx response = httpx.post( "https://api.gov.nominal.io/api/extractors/v2/native/<catalog id>/enable", headers={"Authorization": "Bearer <api key>"}, json={"workspaceRid": "<workspace rid>"}, ) response.raise_for_status() print(response.json())
import asyncio import aiohttp async def main() -> None: async with aiohttp.ClientSession() as session: async with session.post( "https://api.gov.nominal.io/api/extractors/v2/native/<catalog id>/enable", headers={"Authorization": "Bearer <api key>"}, json={"workspaceRid": "<workspace rid>"}, ) as response: response.raise_for_status() print(await response.json()) asyncio.run(main())
Response:
200{ "extractor": { "catalogId": "<catalog id>", "availableVersions": [ "<available versions>" ], "installation": { "rid": "<rid>", "workspaceRid": "<workspace rid>", "name": "<name>", "description": "<description>", "createdAt": "2026-01-01T00:00:00Z", "isArchived": true, "activeContainerImage": { "rid": "<rid>", "tag": "<tag>", "sizeBytes": "1", "status": "CONTAINER_IMAGE_STATUS_PENDING", "createdAt": "2026-01-01T00:00:00Z", "extractorRid": "<extractor rid>", "inputs": [ { "environmentVariable": "<environment variable>", "name": "<name>", "description": "<description>", "fileFilters": [ { "suffix": { "suffix": "<suffix>" } } ], "required": true } ], "parameters": [ { "environmentVariable": "<environment variable>", "name": "<name>", "description": "<description>", "required": true } ], "fileOutputFormat": "FILE_OUTPUT_FORMAT_PARQUET", "defaultTimestampMetadata": { "seriesName": "<series name>", "timestampType": { "relative": { "timeUnit": "<time unit>", "offset": "2026-01-01T00:00:00Z" } } }, "exitCodeMappings": [ { "exitCode": 1, "code": "<code>", "message": "<message>", "retryable": true } ], "resources": { "cpuCores": 1, "memoryGib": 1, "diskGib": 1 } }, "origin": "EXTRACTOR_ORIGIN_CUSTOMER", "catalogId": "<catalog id>", "properties": { "<key>": "<value>" }, "labels": [ "<labels>" ] }, "desiredVersion": "<desired version>", "displayName": "<display name>", "description": "<description>" } }
Types#
AbsoluteTimestamp#
Absolute timestamp parser variants.
Set exactly one of the fields below.
- one of
iso8601,epochOfTimeUnit,customFormatrequired
ContainerImage#
Container image registered through the V2 (extractor-linked) registry flow. V2 images carry no user-supplied
name— extractor identity replaces it as the stable handle.ridstring required
tagstringUser-supplied tag, unique within (workspace, extractor).
sizeBytesstring (int64)Size of the stored image in bytes once the image reaches READY.
statusContainerImageStatusCurrent lifecycle status (PENDING while pushing, READY after a successful push, FAILED otherwise).
createdAtstring (date-time)When the image record was created.
extractorRidstringThe containerized extractor this image is built for.
inputsarray ofFileExtractionInputThe input files this extractor requires, mapped to environment variables that store the path to each file.
parametersarray ofFileExtractionParameterParameters the extractor accepts.
fileOutputFormatFileOutputFormatFormat of the output file produced by the extractor.
defaultTimestampMetadataregistry.v2.TimestampMetadataMetadata about the intermediate parquet this extractor will produce. If not set, timestamp metadata must be provided at ingest time.
exitCodeMappingsarray ofExitCodeMappingFallback exit-code-to-error mappings for extractors that cannot emit a structured /dev/termination-log document.
resourcesContainerResourcesCompute resources this image’s extractor container runs with. Unset fields fall back to the deployment-wide defaults.
ContainerImageStatus#
CONTAINER_IMAGE_STATUS_UNSPECIFIED
CONTAINER_IMAGE_STATUS_PENDING
CONTAINER_IMAGE_STATUS_READY
CONTAINER_IMAGE_STATUS_FAILED
ContainerizedExtractor#
A containerized extractor that processes input files using a self-hosted container image.
Schema fields (inputs, parameters, output format, timestamp metadata) live on the underlying ContainerImage rows; clients should read them off
active_container_image. The extractor itself carries only identity, workspace ownership, user metadata (labels/properties), and lifecycle state.ridstring requiredUnique resource identifier for this extractor.
workspaceRidstring requiredWorkspace that owns this extractor.
namestringUser-supplied name. Need not be unique within a workspace.
descriptionstringFree-form description shown in the UI.
createdAtstring (date-time)Timestamp at which this extractor was first registered.
isArchivedbooleanTrue once the extractor has been archived. Archived extractors are hidden from default search results and reject new ingests; flip back to false on UpdateContainerizedExtractor to restore.
activeContainerImageContainerImageThe container image currently selected to run this extractor. Unset until the first image for this extractor reaches READY status; ingest requests that arrive before then will fail. The schema fields (inputs, parameters, output_file_format, default_timestamp_metadata) live on this nested image.
originExtractorOriginProvenance: CUSTOMER for workspace-authored extractors (the default), FIRST_PARTY for Nominal-published extractors installed from the catalog. Read-only: native-extractor rows are created and upgraded by catalog reconciliation, never through the public write APIs.
catalogIdstringStable catalog identifier (e.g. “ardupilot-dataflash”). Set only when origin is FIRST_PARTY. The version currently active is carried by the installation record and by the active container image’s tag, not duplicated here.
propertiesmap of string to stringFree-form name/value pairs associated with this extractor. At most 64 pairs; names are 1-255 chars and values are up to 1024 chars. At most 64 entries.
labelsarray of stringLabels applied to this extractor, filterable through SearchContainerizedExtractors. At most 64 labels of 1-255 chars each. At most 64 items. Each item: 1 to 255 characters.
ContainerResources#
Compute resources granted to the container this image runs. Every field is optional; an unset field falls back to the deployment-wide default configured on combined-service. The upper bounds are a platform ceiling, not a guarantee the cluster can satisfy the request — an unschedulable pod fails the ingest with EXTRACTOR_UNSCHEDULABLE. RegistryService re-checks them server-side, because the first-party catalog flow calls CreateImage in-process and never reaches protovalidate.
cpuCoresinteger (int64)CPU cores requested for the container. From 1 to 32.
memoryGibinteger (int64)Memory in GiB, applied as both the request and the limit. From 1 to 128.
diskGibinteger (int64)Size in GiB of each of the ephemeral input and output volumes. From 1 to 512.
CustomTimestamp#
Absolute timestamps parsed with a user-supplied format string.
formatstringThe format string should be in the format of the
DateTimeFormatterclass in Java.defaultYearintegerDefault year is accepted as an optional field for cases like IRIG time format. Will be overridden by year in time format.
defaultDayOfYearintegerDefault day of year is accepted as an optional field for cases like IRIG time format. Will be overridden by day of year in time format. From 1 to 366.
EpochTimestamp#
Absolute timestamps expressed as an integer count of some time unit since the Unix epoch.
timeUnitstringUnit in which epoch values are expressed (e.g. SECONDS, MILLISECONDS).
ExitCodeMapping#
Maps a container exit code to a structured error, for extractors that cannot emit a structured /dev/termination-log document. Applied as a fallback error tier at ingest time.
exitCodeintegerThe container process exit code this mapping applies to.
codestring requiredMachine-readable error code. Extractor-defined codes must fall outside the reserved platform namespace.
messagestring requiredHuman-readable error message.
retryablebooleanWhether the platform may retry the ingest for this failure.
ExtractorOrigin#
Provenance of a containerized extractor: authored by a customer workspace, or published by Nominal through the native extractor catalog and materialized into the workspace by catalog reconciliation.
EXTRACTOR_ORIGIN_UNSPECIFIED
EXTRACTOR_ORIGIN_CUSTOMER
EXTRACTOR_ORIGIN_FIRST_PARTY
FileExtractionInput#
Defines an input file to be provided to the extractor.
environmentVariablestring requiredThe environment variable that stores the path to the input file.
namestring requiredName of the input file which users will be prompted with.
descriptionstringDescription of the input file which users will be prompted with.
fileFiltersarray ofFileFilterOptionally filter files for file selection.
requiredbooleanWhether the input file is required for the extractor to run.
FileExtractionParameter#
Defines an input parameter to be provided to the extractor.
environmentVariablestring requiredThe environment variable that stores the argument.
namestring requiredName of the parameter which users will be prompted with.
descriptionstringDescription of the parameter which users will be prompted with.
requiredbooleanWhether the parameter is required for the extractor to run.
FileFilter#
Optional filter applied during file selection for a file input.
suffixFileSuffix
FileOutputFormat#
Output file format emitted by the extractor. Currently PARQUET, CSV, and PARQUET_TAR are the primary supported formats.
FILE_OUTPUT_FORMAT_UNSPECIFIED
FILE_OUTPUT_FORMAT_PARQUET
FILE_OUTPUT_FORMAT_CSV
FILE_OUTPUT_FORMAT_PARQUET_TAR
FILE_OUTPUT_FORMAT_AVRO_STREAM
FILE_OUTPUT_FORMAT_JSON_L
FILE_OUTPUT_FORMAT_MANIFEST
FileSuffix#
The expected suffix of the file. For example, “parquet”, “json”, “csv”, etc.
suffixstring required
Iso8601Timestamp#
Absolute timestamps serialized in ISO-8601 form. No additional parser configuration is needed.
An empty JSON object,
{}.NativeExtractor#
One catalog entry as seen from a workspace: what the deployment carries, plus this workspace’s installation of it (if enabled).
catalogIdstringStable catalog identifier, e.g. “ardupilot-dataflash”.
availableVersionsarray of stringVersions the deployment’s catalog currently carries, ascending.
installationContainerizedExtractorThe workspace-local extractor materialized for this entry, set when enabled. Ingest requests target it exactly like a customer-authored extractor, and the catalog version it is installed at is its active image’s tag.
desiredVersionstringThe single version every workspace on this deployment converges to for this catalog entry. Enable always targets this version; there is no per-workspace pin. Unset only if the deployment has not yet synced this catalog entry.
displayNamestringHuman-readable name from the manifest of the desired version — or of the newest available version when none is desired yet — e.g. “ArduPilot DataFlash”. Enable gives the installation this same name, so an entry in the catalog and the extractor it installs read alike. Falls back to catalog_id when the mirrored manifest cannot be parsed.
descriptionstringFree-form summary from that same manifest, when it declares one.
registry.v2.TimestampMetadata#
Describes how to derive a row timestamp from the extracted output.
seriesNamestring requiredName of the series within the output whose values are used as the row timestamp.
timestampTypeTimestampTyperequiredHow the values in
series_nameshould be interpreted as timestamps.
RelativeTimestamp#
Relative timestamp parser: interprets each row’s timestamp as an offset from a fixed origin.
timeUnitstringUnit in which relative offsets are expressed. Aliases the classic REST TimeUnit alias.
offsetstring (date-time)Starting timestamp to use when indexing the file.
TimestampType#
Selects between relative-offset and absolute-timestamp parsing strategies.
Set exactly one of the fields below.
- one of
relative,absoluterequired
relativeRelativeTimestamp
absoluteAbsoluteTimestamp
Page design based on Scalar, MIT License.
- one of
- one of
- one of
- one of