Dataset retention service#

Manages retention policies and exemptions for datasets.

OpenAPI file: nominal.storage.v1.json

Endpoints#

Upsert dataset retention exemption#

POST /storage/dataset-retention/v1/exemptions

Create or update a retention exemption for a dataset. The exemption is keyed by (dataset_rid, source_id) — upserting with the same key updates the time range.

Body

datasetRid string required

​

sourceId string required

​

bounds Range required

Inclusive time range to exempt from retention.

Fields of Range
start Timestamp

​

Fields of Timestamp
seconds string (int64)

​

nanos string (int64)

​

end Timestamp

​

description string

At most 1,024 characters.

Response

exemption DatasetRetentionExemption

​

Fields of DatasetRetentionExemption
datasetRid string

The dataset this exemption applies to.

sourceId string

Identifier for the source of this exemption (e.g. run RID).

bounds Range

Inclusive time range to exempt from retention.

Fields of Range
start Timestamp

​

Fields of Timestamp
seconds string (int64)

​

nanos string (int64)

​

end Timestamp

​

description string

At most 1,024 characters.

Request

curl -X POST "https://api.gov.nominal.io/api/storage/dataset-retention/v1/exemptions" \
  -H "Authorization: Bearer <api key>" \
  -H "Content-Type: application/json" \
  --data '{
  "datasetRid": "<dataset rid>",
  "sourceId": "<source id>",
  "bounds": {
    "start": {
      "seconds": "1",
      "nanos": "1"
    },
    "end": {
      "seconds": "1",
      "nanos": "1"
    }
  }
}'

import requests

response = requests.post(
    "https://api.gov.nominal.io/api/storage/dataset-retention/v1/exemptions",
    headers={"Authorization": "Bearer <api key>"},
    json={
        "datasetRid": "<dataset rid>",
        "sourceId": "<source id>",
        "bounds": {
            "start": {"seconds": "1", "nanos": "1"},
            "end": {"seconds": "1", "nanos": "1"},
        },
    },
)
response.raise_for_status()
print(response.json())

import httpx

response = httpx.post(
    "https://api.gov.nominal.io/api/storage/dataset-retention/v1/exemptions",
    headers={"Authorization": "Bearer <api key>"},
    json={
        "datasetRid": "<dataset rid>",
        "sourceId": "<source id>",
        "bounds": {
            "start": {"seconds": "1", "nanos": "1"},
            "end": {"seconds": "1", "nanos": "1"},
        },
    },
)
response.raise_for_status()
print(response.json())

import asyncio

import aiohttp


async def main() -> None:
    async with aiohttp.ClientSession() as session:
        async with session.post(
            "https://api.gov.nominal.io/api/storage/dataset-retention/v1/exemptions",
            headers={"Authorization": "Bearer <api key>"},
            json={
                "datasetRid": "<dataset rid>",
                "sourceId": "<source id>",
                "bounds": {
                    "start": {"seconds": "1", "nanos": "1"},
                    "end": {"seconds": "1", "nanos": "1"},
                },
            },
        ) as response:
            response.raise_for_status()
            print(await response.json())


asyncio.run(main())

Response: 200

{
  "exemption": {
    "datasetRid": "<dataset rid>",
    "sourceId": "<source id>",
    "bounds": {
      "start": {
        "seconds": "1",
        "nanos": "1"
      },
      "end": {
        "seconds": "1",
        "nanos": "1"
      }
    },
    "description": "<description>"
  }
}

List dataset retention exemptions#

POST /storage/dataset-retention/v1/exemptions/list

List retention exemptions for a dataset, filtered to those overlapping the given time range.

Body

datasetRid string required

​

bounds Range required

Filter exemptions that overlap with this time range.

Fields of Range
start Timestamp

​

Fields of Timestamp
seconds string (int64)

​

nanos string (int64)

​

end Timestamp

​

pageSize integer

From 1 to 1,000.

nextPageToken string

​

Response

exemptions array of DatasetRetentionExemption

​

Fields of DatasetRetentionExemption
datasetRid string

The dataset this exemption applies to.

sourceId string

Identifier for the source of this exemption (e.g. run RID).

bounds Range

Inclusive time range to exempt from retention.

Fields of Range
start Timestamp

​

Fields of Timestamp
seconds string (int64)

​

nanos string (int64)

​

end Timestamp

​

description string

At most 1,024 characters.

nextPageToken string

​

Request

curl -X POST "https://api.gov.nominal.io/api/storage/dataset-retention/v1/exemptions/list" \
  -H "Authorization: Bearer <api key>" \
  -H "Content-Type: application/json" \
  --data '{
  "datasetRid": "<dataset rid>",
  "bounds": {
    "start": {
      "seconds": "1",
      "nanos": "1"
    },
    "end": {
      "seconds": "1",
      "nanos": "1"
    }
  }
}'

import requests

response = requests.post(
    "https://api.gov.nominal.io/api/storage/dataset-retention/v1/exemptions/list",
    headers={"Authorization": "Bearer <api key>"},
    json={
        "datasetRid": "<dataset rid>",
        "bounds": {
            "start": {"seconds": "1", "nanos": "1"},
            "end": {"seconds": "1", "nanos": "1"},
        },
    },
)
response.raise_for_status()
print(response.json())

import httpx

response = httpx.post(
    "https://api.gov.nominal.io/api/storage/dataset-retention/v1/exemptions/list",
    headers={"Authorization": "Bearer <api key>"},
    json={
        "datasetRid": "<dataset rid>",
        "bounds": {
            "start": {"seconds": "1", "nanos": "1"},
            "end": {"seconds": "1", "nanos": "1"},
        },
    },
)
response.raise_for_status()
print(response.json())

import asyncio

import aiohttp


async def main() -> None:
    async with aiohttp.ClientSession() as session:
        async with session.post(
            "https://api.gov.nominal.io/api/storage/dataset-retention/v1/exemptions/list",
            headers={"Authorization": "Bearer <api key>"},
            json={
                "datasetRid": "<dataset rid>",
                "bounds": {
                    "start": {"seconds": "1", "nanos": "1"},
                    "end": {"seconds": "1", "nanos": "1"},
                },
            },
        ) as response:
            response.raise_for_status()
            print(await response.json())


asyncio.run(main())

Response: 200

{
  "exemptions": [
    {
      "datasetRid": "<dataset rid>",
      "sourceId": "<source id>",
      "bounds": {
        "start": {
          "seconds": "1",
          "nanos": "1"
        },
        "end": {
          "seconds": "1",
          "nanos": "1"
        }
      },
      "description": "<description>"
    }
  ],
  "nextPageToken": "<next page token>"
}

Delete dataset retention exemption#

DELETE /storage/dataset-retention/v1/exemptions/{datasetRid}/{sourceId}

Delete a retention exemption for a dataset.

Path parameters

datasetRid string required

​

sourceId string required

​

Response

An empty JSON object, {}.

Request

curl -X DELETE "https://api.gov.nominal.io/api/storage/dataset-retention/v1/exemptions/<dataset rid>/<source id>" \
  -H "Authorization: Bearer <api key>"

import requests

response = requests.delete(
    "https://api.gov.nominal.io/api/storage/dataset-retention/v1/exemptions/<dataset rid>/<source id>",
    headers={"Authorization": "Bearer <api key>"},
)
response.raise_for_status()
print(response.json())

import httpx

response = httpx.delete(
    "https://api.gov.nominal.io/api/storage/dataset-retention/v1/exemptions/<dataset rid>/<source id>",
    headers={"Authorization": "Bearer <api key>"},
)
response.raise_for_status()
print(response.json())

import asyncio

import aiohttp


async def main() -> None:
    async with aiohttp.ClientSession() as session:
        async with session.delete(
            "https://api.gov.nominal.io/api/storage/dataset-retention/v1/exemptions/<dataset rid>/<source id>",
            headers={"Authorization": "Bearer <api key>"},
        ) as response:
            response.raise_for_status()
            print(await response.json())


asyncio.run(main())

Response: 200

{}

Update retention policy#

PUT /storage/dataset-retention/v1/policy/{datasetRid}

Update or remove the retention policy for a dataset. When a policy is set, data older than the policy’s expiry will be deleted.

Path parameters

datasetRid string required

​

Body

retentionPolicy RetentionPolicy

If absent, the retention policy is removed (reverts to default/keep-forever).

Fields of RetentionPolicy
one of keepForever, keepUntilExpiry required

​

keepForever KeepForever

​

keepUntilExpiry KeepUntilExpiry

​

Fields of KeepUntilExpiry
expiryDays integer

Greater than 0.

Response

An empty JSON object, {}.

Request

curl -X PUT "https://api.gov.nominal.io/api/storage/dataset-retention/v1/policy/<dataset rid>" \
  -H "Authorization: Bearer <api key>" \
  -H "Content-Type: application/json" \
  --data '{
  "retentionPolicy": {
    "keepForever": {}
  }
}'

import requests

response = requests.put(
    "https://api.gov.nominal.io/api/storage/dataset-retention/v1/policy/<dataset rid>",
    headers={"Authorization": "Bearer <api key>"},
    json={"retentionPolicy": {"keepForever": {}}},
)
response.raise_for_status()
print(response.json())

import httpx

response = httpx.put(
    "https://api.gov.nominal.io/api/storage/dataset-retention/v1/policy/<dataset rid>",
    headers={"Authorization": "Bearer <api key>"},
    json={"retentionPolicy": {"keepForever": {}}},
)
response.raise_for_status()
print(response.json())

import asyncio

import aiohttp


async def main() -> None:
    async with aiohttp.ClientSession() as session:
        async with session.put(
            "https://api.gov.nominal.io/api/storage/dataset-retention/v1/policy/<dataset rid>",
            headers={"Authorization": "Bearer <api key>"},
            json={"retentionPolicy": {"keepForever": {}}},
        ) as response:
            response.raise_for_status()
            print(await response.json())


asyncio.run(main())

Response: 200

{}

Get dataset retention watermark#

GET /storage/dataset-retention/v1/watermarks/{datasetRid}

Get the retention watermark for a dataset.

Path parameters

datasetRid string required

​

Response

watermark Timestamp

​

Fields of Timestamp
seconds string (int64)

​

nanos string (int64)

​

Request

curl "https://api.gov.nominal.io/api/storage/dataset-retention/v1/watermarks/<dataset rid>" \
  -H "Authorization: Bearer <api key>"

import requests

response = requests.get(
    "https://api.gov.nominal.io/api/storage/dataset-retention/v1/watermarks/<dataset rid>",
    headers={"Authorization": "Bearer <api key>"},
)
response.raise_for_status()
print(response.json())

import httpx

response = httpx.get(
    "https://api.gov.nominal.io/api/storage/dataset-retention/v1/watermarks/<dataset rid>",
    headers={"Authorization": "Bearer <api key>"},
)
response.raise_for_status()
print(response.json())

import asyncio

import aiohttp


async def main() -> None:
    async with aiohttp.ClientSession() as session:
        async with session.get(
            "https://api.gov.nominal.io/api/storage/dataset-retention/v1/watermarks/<dataset rid>",
            headers={"Authorization": "Bearer <api key>"},
        ) as response:
            response.raise_for_status()
            print(await response.json())


asyncio.run(main())

Response: 200

{
  "watermark": {
    "seconds": "1",
    "nanos": "1"
  }
}

Update dataset retention watermark#

PUT /storage/dataset-retention/v1/watermarks/{datasetRid}

Update the retention watermark for a dataset.

Path parameters

datasetRid string required

​

Body

watermark Timestamp required

​

Fields of Timestamp
seconds string (int64)

​

nanos string (int64)

​

Response

watermark Timestamp

​

Fields of Timestamp
seconds string (int64)

​

nanos string (int64)

​

Request

curl -X PUT "https://api.gov.nominal.io/api/storage/dataset-retention/v1/watermarks/<dataset rid>" \
  -H "Authorization: Bearer <api key>" \
  -H "Content-Type: application/json" \
  --data '{
  "watermark": {
    "seconds": "1",
    "nanos": "1"
  }
}'

import requests

response = requests.put(
    "https://api.gov.nominal.io/api/storage/dataset-retention/v1/watermarks/<dataset rid>",
    headers={"Authorization": "Bearer <api key>"},
    json={"watermark": {"seconds": "1", "nanos": "1"}},
)
response.raise_for_status()
print(response.json())

import httpx

response = httpx.put(
    "https://api.gov.nominal.io/api/storage/dataset-retention/v1/watermarks/<dataset rid>",
    headers={"Authorization": "Bearer <api key>"},
    json={"watermark": {"seconds": "1", "nanos": "1"}},
)
response.raise_for_status()
print(response.json())

import asyncio

import aiohttp


async def main() -> None:
    async with aiohttp.ClientSession() as session:
        async with session.put(
            "https://api.gov.nominal.io/api/storage/dataset-retention/v1/watermarks/<dataset rid>",
            headers={"Authorization": "Bearer <api key>"},
            json={"watermark": {"seconds": "1", "nanos": "1"}},
        ) as response:
            response.raise_for_status()
            print(await response.json())


asyncio.run(main())

Response: 200

{
  "watermark": {
    "seconds": "1",
    "nanos": "1"
  }
}

Types#

DatasetRetentionExemption#

A retention exemption for a dataset, representing a time range that should be preserved regardless of the dataset’s retention policy.

datasetRid string required

The dataset this exemption applies to.

sourceId string required

Identifier for the source of this exemption (e.g. run RID).

bounds Range required

Inclusive time range to exempt from retention.

description string

At most 1,024 characters.

KeepForever#

An empty JSON object, {}.

KeepUntilExpiry#

expiryDays integer

Greater than 0.

Range#

Deprecated for new APIs. Define an API-owned range with the endpoint semantics it needs instead.

start Timestamp

​

end Timestamp

​

RetentionPolicy#

Set exactly one of the fields below.

one of keepForever, keepUntilExpiry required

​

keepForever KeepForever

​

keepUntilExpiry KeepUntilExpiry

​

Timestamp#

Deprecated for new APIs. Use google.protobuf.Timestamp for absolute instants.

This legacy nanosecond-precision timestamp is represented by an epoch time in seconds and a nanosecond offset. The nanosecond offset is from the start of the epoch second, so must be less than 1 billion.

seconds string (int64)

​

nanos string (int64)

​

Page design based on Scalar, MIT License.