Upload service#

The Upload Service manages file uploads to object storage.

OpenAPI file: UploadService.json

Endpoints#

Initiate multipart upload#

POST /upload/v1/multipart-upload

Initiates a multipart upload to object storage. Returns an uploadId that should be used with listParts, signPart, and completeMultipartUpload.

Body

filename string required

The desired name of the file in object storage. The final name will be prefixed with a timestamp to ensure uniqueness.

filetype string required

The MIME type of the file, eg “text/csv” for CSVs or “application/octet-stream” for binary files.

workspace string (RID)

The workspace in which to upload the file. If not provided, the file will be uploaded to the default workspace for the user’s organization, if that is configured and the user has access to it. Ignored if resource is provided.

resource string (RID)

The RID of an existing resource (e.g. a dataset) that this upload is intended to feed. When provided, the server resolves the destination workspace from the resource and authorizes the caller via the resource’s permissions rather than via the workspace default-fallback. Use this in flows where the caller already knows the target resource (e.g. mesh file ingest into an existing remote dataset); this avoids the Security:WorkspaceNotSpecified failure for API keys whose user has no default workspace. If both resource and workspace are set, resource takes precedence.

destination UploadDestination

Where the upload should be written. Defaults to UPLOADS. Whichever destination is chosen, the returned bucket must be passed back on the listParts/signPart/completeMultipartUpload/abortMultipartUpload calls.

Values of UploadDestination
UPLOADS

​

FILE_STORE

​

Response

uploadId string

​

key string

Identifies the uploaded object within its destination. For a FILE_STORE upload this is the value to pass as UploadedObjectRef.objectKey when committing a file revision.

bucket string

Opaque handle for the upload’s destination. Pass it back unchanged on every subsequent call for this upload; do not parse or display it.

Request

curl -X POST "https://api.gov.nominal.io/api/upload/v1/multipart-upload" \
  -H "Authorization: Bearer <api key>" \
  -H "Content-Type: application/json" \
  --data '{
  "filename": "<filename>",
  "filetype": "<filetype>"
}'

import requests

response = requests.post(
    "https://api.gov.nominal.io/api/upload/v1/multipart-upload",
    headers={"Authorization": "Bearer <api key>"},
    json={"filename": "<filename>", "filetype": "<filetype>"},
)
response.raise_for_status()
print(response.json())

import httpx

response = httpx.post(
    "https://api.gov.nominal.io/api/upload/v1/multipart-upload",
    headers={"Authorization": "Bearer <api key>"},
    json={"filename": "<filename>", "filetype": "<filetype>"},
)
response.raise_for_status()
print(response.json())

import asyncio

import aiohttp


async def main() -> None:
    async with aiohttp.ClientSession() as session:
        async with session.post(
            "https://api.gov.nominal.io/api/upload/v1/multipart-upload",
            headers={"Authorization": "Bearer <api key>"},
            json={"filename": "<filename>", "filetype": "<filetype>"},
        ) as response:
            response.raise_for_status()
            print(await response.json())


asyncio.run(main())

Response: 200

{
  "uploadId": "<upload id>",
  "key": "<key>",
  "bucket": "<bucket>"
}

List parts#

GET /upload/v1/multipart-upload/{uploadId}

Lists the parts that have been uploaded for a given uploadId.

Path parameters

uploadId string required

​

Query parameters

key string required

​

bucket string

The opaque destination handle returned as bucket from InitiateMultipartUploadResponse, passed back unchanged. Omit only for an upload opened without a destination.

Response

The body is an array of PartWithSize.

Fields of PartWithSize
partNumber integer

​

etag string

​

size integer (int64)

​

Request

curl "https://api.gov.nominal.io/api/upload/v1/multipart-upload/<upload id>?key=<key>" \
  -H "Authorization: Bearer <api key>"

import requests

response = requests.get(
    "https://api.gov.nominal.io/api/upload/v1/multipart-upload/<upload id>?key=<key>",
    headers={"Authorization": "Bearer <api key>"},
)
response.raise_for_status()
print(response.json())

import httpx

response = httpx.get(
    "https://api.gov.nominal.io/api/upload/v1/multipart-upload/<upload id>?key=<key>",
    headers={"Authorization": "Bearer <api key>"},
)
response.raise_for_status()
print(response.json())

import asyncio

import aiohttp


async def main() -> None:
    async with aiohttp.ClientSession() as session:
        async with session.get(
            "https://api.gov.nominal.io/api/upload/v1/multipart-upload/<upload id>?key=<key>",
            headers={"Authorization": "Bearer <api key>"},
        ) as response:
            response.raise_for_status()
            print(await response.json())


asyncio.run(main())

Response: 200

[
  {
    "partNumber": 1,
    "etag": "<etag>",
    "size": 1
  }
]

Sign part#

POST /upload/v1/multipart-upload/{uploadId}

Signs an upload request for a single part. Returns a URL that will execute the upload without further authentication.

Path parameters

uploadId string required

​

Query parameters

key string required

​

partNumber integer required

​

bucket string

The opaque destination handle returned as bucket from InitiateMultipartUploadResponse, passed back unchanged. Omit only for an upload opened without a destination.

Response

url string

​

headers map of string to string

​

expiresInSeconds integer (int64)

How long the signed URL stays valid, from when it was issued. Relative rather than absolute so a client with a skewed clock still bounds the request correctly; clients should time the upload out at this point instead of letting a request run past the signature’s lifetime and fail as an opaque connection reset.

Request

curl -X POST "https://api.gov.nominal.io/api/upload/v1/multipart-upload/<upload id>?key=<key>&partNumber=1" \
  -H "Authorization: Bearer <api key>"

import requests

response = requests.post(
    "https://api.gov.nominal.io/api/upload/v1/multipart-upload/<upload id>?key=<key>&partNumber=1",
    headers={"Authorization": "Bearer <api key>"},
)
response.raise_for_status()
print(response.json())

import httpx

response = httpx.post(
    "https://api.gov.nominal.io/api/upload/v1/multipart-upload/<upload id>?key=<key>&partNumber=1",
    headers={"Authorization": "Bearer <api key>"},
)
response.raise_for_status()
print(response.json())

import asyncio

import aiohttp


async def main() -> None:
    async with aiohttp.ClientSession() as session:
        async with session.post(
            "https://api.gov.nominal.io/api/upload/v1/multipart-upload/<upload id>?key=<key>&partNumber=1",
            headers={"Authorization": "Bearer <api key>"},
        ) as response:
            response.raise_for_status()
            print(await response.json())


asyncio.run(main())

Response: 200

{
  "url": "<url>",
  "headers": {
    "<key>": "<value>"
  },
  "expiresInSeconds": 1
}

Complete multipart upload#

POST /upload/v1/multipart-upload/{uploadId}/complete

Completes a multipart upload to object storage. This should be called after all parts have been uploaded. Will throw EmptyMultipartUpload if there are 0 parts.

Path parameters

uploadId string required

​

Query parameters

key string required

​

bucket string

The opaque destination handle returned as bucket from InitiateMultipartUploadResponse, passed back unchanged. Omit only for an upload opened without a destination.

Body

The body is an array of Part.

Fields of Part
partNumber integer required

​

etag string required

​

Response

location string

​

Request

curl -X POST "https://api.gov.nominal.io/api/upload/v1/multipart-upload/<upload id>/complete?key=<key>" \
  -H "Authorization: Bearer <api key>" \
  -H "Content-Type: application/json" \
  --data '[
  {
    "partNumber": 1,
    "etag": "<etag>"
  }
]'

import requests

response = requests.post(
    "https://api.gov.nominal.io/api/upload/v1/multipart-upload/<upload id>/complete?key=<key>",
    headers={"Authorization": "Bearer <api key>"},
    json=[{"partNumber": 1, "etag": "<etag>"}],
)
response.raise_for_status()
print(response.json())

import httpx

response = httpx.post(
    "https://api.gov.nominal.io/api/upload/v1/multipart-upload/<upload id>/complete?key=<key>",
    headers={"Authorization": "Bearer <api key>"},
    json=[{"partNumber": 1, "etag": "<etag>"}],
)
response.raise_for_status()
print(response.json())

import asyncio

import aiohttp


async def main() -> None:
    async with aiohttp.ClientSession() as session:
        async with session.post(
            "https://api.gov.nominal.io/api/upload/v1/multipart-upload/<upload id>/complete?key=<key>",
            headers={"Authorization": "Bearer <api key>"},
            json=[{"partNumber": 1, "etag": "<etag>"}],
        ) as response:
            response.raise_for_status()
            print(await response.json())


asyncio.run(main())

Response: 200

{
  "location": "<location>"
}

Abort multipart upload#

POST /upload/v1/multipart-upload/{uploadId}/abort

Aborts a multipart upload to S3. Frees storage used by previously uploaded parts and prevents further uploads to the same uploadId.

Path parameters

uploadId string required

​

Query parameters

key string required

​

bucket string

The opaque destination handle returned as bucket from InitiateMultipartUploadResponse, passed back unchanged. Omit only for an upload opened without a destination.

Response

No content (HTTP 204).

Request

curl -X POST "https://api.gov.nominal.io/api/upload/v1/multipart-upload/<upload id>/abort?key=<key>" \
  -H "Authorization: Bearer <api key>"

import requests

response = requests.post(
    "https://api.gov.nominal.io/api/upload/v1/multipart-upload/<upload id>/abort?key=<key>",
    headers={"Authorization": "Bearer <api key>"},
)
response.raise_for_status()

import httpx

response = httpx.post(
    "https://api.gov.nominal.io/api/upload/v1/multipart-upload/<upload id>/abort?key=<key>",
    headers={"Authorization": "Bearer <api key>"},
)
response.raise_for_status()

import asyncio

import aiohttp


async def main() -> None:
    async with aiohttp.ClientSession() as session:
        async with session.post(
            "https://api.gov.nominal.io/api/upload/v1/multipart-upload/<upload id>/abort?key=<key>",
            headers={"Authorization": "Bearer <api key>"},
        ) as response:
            response.raise_for_status()


asyncio.run(main())

Response: 204

No content.

Sign download#

POST /upload/v1/sign-download

Returns a short-lived signed GET URL for an object already in the uploads bucket. Use this to hand a freshly-uploaded object’s path to an external service (e.g. Dagger) without giving that service any Nominal credentials. The caller must be authorized to upload to the workspace that owns the path (encoded as the first segment of the object key).

Body

path string required

Path of an object in the Nominal uploads bucket, in the form returned by CompleteMultipartUploadResponse.location. The caller must be authorized to read the workspace whose UUID prefixes the object key.

Response

url string

Short-lived signed GET URL. Expires after 15 minutes.

expiresAt string (date-time)

Absolute expiry timestamp. Callers must stop using the URL after this.

Request

curl -X POST "https://api.gov.nominal.io/api/upload/v1/sign-download" \
  -H "Authorization: Bearer <api key>" \
  -H "Content-Type: application/json" \
  --data '{
  "path": "<s3 path>"
}'

import requests

response = requests.post(
    "https://api.gov.nominal.io/api/upload/v1/sign-download",
    headers={"Authorization": "Bearer <api key>"},
    json={"path": "<s3 path>"},
)
response.raise_for_status()
print(response.json())

import httpx

response = httpx.post(
    "https://api.gov.nominal.io/api/upload/v1/sign-download",
    headers={"Authorization": "Bearer <api key>"},
    json={"path": "<s3 path>"},
)
response.raise_for_status()
print(response.json())

import asyncio

import aiohttp


async def main() -> None:
    async with aiohttp.ClientSession() as session:
        async with session.post(
            "https://api.gov.nominal.io/api/upload/v1/sign-download",
            headers={"Authorization": "Bearer <api key>"},
            json={"path": "<s3 path>"},
        ) as response:
            response.raise_for_status()
            print(await response.json())


asyncio.run(main())

Response: 200

{
  "url": "<url>",
  "expiresAt": "2026-01-01T00:00:00Z"
}

Upload file#

POST /upload/v1/upload-file

Uploads a file to S3. Intended for smaller files.

Query parameters

fileName string required

​

sizeBytes integer (int64)

Specifies the size in bytes of the file being uploaded. This is required to be set if the file is greater than 100MB, however in practice this should probably always be set.

workspace string (RID)

The workspace in which to upload the file. If not provided, the file will be uploaded to the default workspace for the user’s organization, if that is configured and the user has access to it.

Body: application/octet-stream

The body is the raw bytes.

Response

The body is a string.

Request

curl -X POST "https://api.gov.nominal.io/api/upload/v1/upload-file?fileName=<file name>" \
  -H "Authorization: Bearer <api key>" \
  -H "Content-Type: application/octet-stream" \
  --data-binary "@<file>"

from pathlib import Path

import requests

response = requests.post(
    "https://api.gov.nominal.io/api/upload/v1/upload-file?fileName=<file name>",
    headers={
        "Authorization": "Bearer <api key>",
        "Content-Type": "application/octet-stream",
    },
    data=Path("<file>").read_bytes(),
)
response.raise_for_status()
print(response.json())

from pathlib import Path

import httpx

response = httpx.post(
    "https://api.gov.nominal.io/api/upload/v1/upload-file?fileName=<file name>",
    headers={
        "Authorization": "Bearer <api key>",
        "Content-Type": "application/octet-stream",
    },
    content=Path("<file>").read_bytes(),
)
response.raise_for_status()
print(response.json())

import asyncio
from pathlib import Path

import aiohttp


async def main() -> None:
    async with aiohttp.ClientSession() as session:
        async with session.post(
            "https://api.gov.nominal.io/api/upload/v1/upload-file?fileName=<file name>",
            headers={
                "Authorization": "Bearer <api key>",
                "Content-Type": "application/octet-stream",
            },
            data=Path("<file>").read_bytes(),
        ) as response:
            response.raise_for_status()
            print(await response.json())


asyncio.run(main())

Response: 200

"<s3 path>"

Errors#

Errors shows how these errors are sent.

Each error’s errorName is UploadService: followed by its name below.

Status

Name

Notes

400

EmptyMultipartUpload

400

IncorrectSizeBytes

400

NullKey

400

SignDownloadPathOutsideUploadsBucket

400

UploadBucketNotAllowed

A multipart-upload follow-up call specified a bucket this service does not recognize. It must be the value returned as bucket by initiateMultipartUpload, unchanged.

400

WorkspaceResourceMismatch

Both workspace and resource were set on the upload request, but the workspace derived from the resource does not match the explicitly-supplied workspace. Returned as a defensive sanity check so callers don’t accidentally believe they’re uploading to one workspace while the file is actually being routed to another.

403

SignDownloadUnauthorized

404

ResourceNotFoundOrUnauthorized

The resource referenced by an upload’s resource field was not found or the caller is not authorized to access it. Returned as NOT_FOUND (rather than PERMISSION_DENIED) to avoid leaking existence information about resources the caller cannot see.

404

UploadObjectNotFound

A multipart-upload follow-up call named a File Store object the ledger has no row for. A malformed key and an unknown one are reported identically, so a caller cannot tell which of the two it supplied.

Each error’s errorName is IngestService: followed by its name below.

Status

Name

Notes

400

BatchGranularityMismatch

All dataset granularities must match.

400

ContainerizedExtractorTimestampMetadataMissing

Neither the containerized extractor (or its active container image) nor the ingest request provides timestamp metadata, so the extractor output cannot be indexed. Register the image with default timestamp metadata or supply it on the request.

400

FailedToDownloadFileUserError

400

FileDeletionNotSupported

The dataset file is part of a multi-file ingest job and cannot be deleted on its own. Delete the entire ingest job instead.

400

FrameCountTimestampsMismatch

400

IncompatibleConnectionTypeAndLogQuery

400

IncompatibleFileForDataset

400

IncompatibleInputs

The inputs provided to the ingest job are incompatible with each other

400

IngestJobNotCompatibleForRerun

The ingest job exists but is not compatible to be re-run.

400

InvalidBucket

400

InvalidCustomTimestampFormat

400

InvalidS3Path

400

InvalidUrl

400

MissingMetadataForReingest

The timestamp metadata required to reingest the dataset is missing and must be provided in the request.

400

ParallelReingestIntoDatasetV1

Cannot ingest multiple datasets in parallel into older datasets. It is recommended to reingest all source datasets and the target dataset into a new dataset destination.

400

RecursiveReingest

The target dataset is included in the source datasets.

400

ReingestTooManyFiles

The number of files to reingest is too large.

400

RunBoundsInverted

400

UnsupportedIngestSource

400

UnsupportedRequestTypeForReIngest

403

ContainerizedExtractorUnauthorizedPasswordAccess

The user is not authorized to access the containerized extractor password.

404

DatasetFileNotFound

The target dataset file could not be found.

404

DatasetNotFound

The target dataset could not be found.

404

IngestJobNotFound

404

IngestTransformNotFound

The ingest transform does not exist, or its parent ingest job is not visible to the caller. Returned as NOT_FOUND (rather than PERMISSION_DENIED) to avoid leaking existence information about transforms the caller cannot see.

404

S3PathNotFound

404

StreamingSessionNotFound

The streaming session could not be found.

409

IngestJobNotCancellable

The ingest job exists but is in a terminal state (COMPLETED, FAILED, or CANCELLED) and therefore cannot be cancelled.

500

ContainerizedExtractorContainerFailed

500

FailedToDownloadFile

500

IngestJobMissingDatasetForIdempotencyKey

A prior ingest job already exists for this idempotency key but never produced a dataset (the original submit did not get far enough). The idempotency key deduplicates to that existing job and does not re-run ingestion; rerun the job identified by ingestJobRid to complete it.

Types#

Part#

partNumber integer required

​

etag string required

​

PartWithSize#

partNumber integer required

​

etag string required

​

size integer (int64) required

​

UploadDestination#

The object-storage destination an upload should be written to.

UPLOADS

​

FILE_STORE

​

Page design based on Scalar, MIT License.