Export jobs (gRPC)#
Write query results to files in object storage, and download them when the export finishes. An export runs in the background, so you start it, poll it, and then download its files.
The service is ExportJobService in the nominal.exportjobs.v1 proto package. It has three calls:
Call |
Purpose |
|---|---|
|
Validate and authorize the request, then start the export. |
|
Return one export, with a download URL for each file once it is complete. |
|
List a workspace’s exports, newest first. |
Connect and authenticate as described in the gRPC API overview. Every call needs view access to the export’s workspace. A call without access gets the same NOT_FOUND as a call for an export that doesn’t exist.
Start an export#
CreateExportJob takes a workspace, one source, and optional settings.
Field |
Type |
Required |
Notes |
|---|---|---|---|
|
string |
Yes |
The workspace the export runs in. |
|
|
One source is required |
|
|
|
One source is required |
|
|
|
One source is required |
|
|
|
No |
Must be unset for a |
sql, compute and zip are alternatives: set exactly one.
The response holds an ExportJob. Keep its export_id, which is the id to pass to GetExportJob. The call returns as soon as the export is started, so status is usually EXPORT_JOB_STATUS_SUBMITTED or EXPORT_JOB_STATUS_IN_PROGRESS, though a very fast export can already be complete.
An SQL export, shown as JSON:
{
"workspace_rid": "<workspace-rid>",
"sql": {
"query": "SELECT ts, value FROM points_double WHERE dataset_rid = '<dataset-rid>' ORDER BY ts"
},
"settings": {
"file_format": "EXPORT_FILE_FORMAT_PARQUET"
}
}
A zip export of every file in one dataset, and of two files in another:
{
"workspace_rid": "<workspace-rid>",
"zip": {
"selections": [
{ "dataset_rid": "<dataset-rid>", "all_files": {} },
{ "dataset_rid": "<other-dataset-rid>", "file_ids": { "ids": ["<file-uuid>", "<file-uuid>"] } }
]
}
}
Sources#
SQL: runs the query as the SQL export endpoint does. The caller needs the access that endpoint requires.
Compute: exports computed channel data. The caller needs view access to the workspace and access to the data the definition reads. The server checks the definition again when the export runs.
Zip: puts dataset files in one zip. Each
DatasetFilesSelectionnames adataset_ridand exactly one offile_ids(1 to 1,000 file UUIDs, zipped in the order given),query(every uploaded, non-deleted file that matches a dataset file query) orall_files(every uploaded, non-deleted file). The caller needs read access to every dataset named. The selections are resolved when the export runs, not when you create it.
Settings#
settings is the state of an export form. The server stores it and returns it as given, so a client can prefill a new export from an earlier one.
For an
sqlexport, the server uses onlyfile_formatandparquet_options.file_formatisEXPORT_FILE_FORMAT_CSV(also the default when unset) orEXPORT_FILE_FORMAT_PARQUET.parquet_optionsis valid only with Parquet.For a
computeexport, the server uses none of the settings. Thedefinitiondecides what is exported, and isn’t checked againstsettings.A
zipexport takes no settings, and a request that sets them is rejected withINVALID_ARGUMENT.
Get an export#
GetExportJob takes the export_id and returns an ExportJob:
Field |
Notes |
|---|---|
|
The id to pass to |
|
The settings sent at creation, if any. |
|
An |
|
The exported files. Set only when |
|
When the export was requested. |
Each ExportFile has a file_name, size_bytes, expire_time, and, from GetExportJob only, a download_url and its download_url_expire_time. The URL is a presigned GET URL that carries its own credentials, so treat it as a secret and don’t log it. Each call to GetExportJob returns a new URL. When one expires, call it again.
An export can be seen by the user who created it, and by other users who can view the workspace once the export is complete and they can read every dataset it read. An export that is still running is visible only to its creator.
List exports#
ListExportJobs returns the exports in a workspace that you can see, newest first.
Field |
Type |
Required |
Notes |
|---|---|---|---|
|
string |
Yes |
The workspace to list. A workspace you can’t view lists nothing. |
|
integer |
No |
1 to 1,000. Defaults to 50. |
|
string |
No |
The |
The response has export_jobs and, when there are more, next_page_token. Each entry is an ExportJobSummary with the export_id, status and create_time, but not the settings or files. Call GetExportJob for those. A page can hold fewer than page_size exports when some are left out because you can’t read their datasets, so continue until next_page_token is unset.
Export lifecycle#
Poll GetExportJob until the status is EXPORT_JOB_STATUS_COMPLETED, EXPORT_JOB_STATUS_FAILED or EXPORT_JOB_STATUS_CANCELLED.
Status |
Meaning |
|---|---|
|
Recorded, but not yet submitted to the job service. |
|
Running. |
|
Done. |
|
Failed. |
|
Cancelled. |
|
Completed, but its files are past their retention period and no longer available. |
Files are kept for 7 days after the export completes, after which the export reports EXPORT_JOB_STATUS_EXPIRED.
Errors#
A failed call returns a gRPC status. For the errors this service defines, the status has a google.rpc.ErrorInfo detail whose reason is one of these, as described in Errors:
|
Status code |
When |
|---|---|---|
|
|
The export doesn’t exist, or you can’t see it. The two look the same. |
|
|
|
|
|
The deployment has no object storage for exports. |
|
|
A zip request names more files than one export may. |
|
|
A zip request names a dataset or file that is missing, in another workspace, deleted, or unreadable by you. These look the same. |
CreateExportJob also returns NOT_FOUND when the workspace rid is malformed or you can’t view the workspace, and INVALID_ARGUMENT for a request that fails validation, such as a missing source or a zip request with settings.