Download OpenAPI specification:Download
Marquez is an open source metadata service for the collection, aggregation, and visualization of a data ecosystem's metadata.
Creates a new namespace object. A namespace enables the contextual grouping of related jobs and datasets. Namespaces must contain only letters (a-z
, A-Z
), numbers (0-9
), underscores (_
), dashes (-
), colons (:
), slashes (/
), or dots (.
). A namespace is case-insensitive with a maximum length of 1024
characters. Note jobs and datasets will be unique within a namespace, but not across namespaces.
namespace required | string <= 1024 characters Example: my-namespace The name of the namespace. |
ownerName required | string The owner of the namespace. |
description | string The description of the namespace. |
{- "ownerName": "me",
- "description": "My first namespace!"
}
{- "name": "my-namespace",
- "createdAt": "2019-05-09T19:49:24.201361Z",
- "updatedAt": "2019-05-09T19:49:24.201361Z",
- "ownerName": "me",
- "description": "My first namespace!"
}
Returns a namespace.
namespace required | string <= 1024 characters Example: my-namespace The name of the namespace. |
{- "name": "my-namespace",
- "createdAt": "2019-05-09T19:49:24.201361Z",
- "updatedAt": "2019-05-09T19:49:24.201361Z",
- "ownerName": "me",
- "description": "My first namespace!"
}
Soft deletes a namespace, and every job and dataset inside. On next event containing this namespace, the namespace will be undeleted.
namespace required | string <= 1024 characters Example: my-namespace The name of the namespace. |
{- "name": "my-namespace",
- "createdAt": "2019-05-09T19:49:24.201361Z",
- "updatedAt": "2019-05-09T19:49:24.201361Z",
- "ownerName": "me",
- "description": "My first namespace!"
}
Returns a list of namespaces.
limit | integer Default: 100 Example: limit=25 The number of results to return from offset. |
offset | integer Default: 0 The initial position from which to return results. |
{- "namespaces": [
- {
- "name": "my-namespace",
- "createdAt": "2019-05-09T19:49:24.201361Z",
- "updatedAt": "2019-05-09T19:49:24.201361Z",
- "ownerName": "me",
- "description": "My first namespace!"
}
]
}
Returns a list of OpenLineage events, sorted in direction of passed sort parameter. By default it is desc.
sortDirection | string Example: sortDirection=name Sorts the results of your query by indicated direction |
before | string <date-time> Example: before=2022-09-15T07:47:19Z Returns events before passed date. |
after | string <date-time> Example: after=2022-09-15T07:47:19Z Returns events after passed date. |
limit | integer Default: 100 Example: limit=25 The number of results to return from offset. |
offset | integer Default: 0 The initial position from which to return results. |
{- "events": [
- {
- "eventType": "COMPLETE",
- "eventTime": "2019-05-09T19:49:24.201361Z",
- "run": {
- "runId": "d46e465b-d358-4d32-83d4-df660ff614dd"
}, - "job": {
- "namespace": "my-namespace",
- "name": "my-job"
}, - "outputs": [
- {
- "namespace": "my-namespace",
- "name": "my-output",
- "facets": {
- "schema": {
- "fields": [
- {
- "name": "a",
- "type": "INTEGER"
}, - {
- "name": "b",
- "type": "TIMESTAMP"
}, - {
- "name": "c",
- "type": "INTEGER"
}, - {
- "name": "d",
- "type": "INTEGER"
}
]
}
}
}
],
}
], - "totalCount": 0
}
Creates a new source object. A source is the physical location of a dataset such as a table in PostgreSQL, or topic in Kafka. A source enables the grouping of physical datasets to their physical source.
source required | string <= 1024 characters Example: my-source The name of the source. |
type required | string The type of the source. |
connectionUrl required | string <URL> The URL to the location of the source. |
description | string The description of the source. |
{- "type": "POSTGRESQL",
- "connectionUrl": "jdbc:postgresql://db.example.com/mydb",
- "description": "My first source!"
}
{- "type": "POSTGRESQL",
- "name": "my-source",
- "createdAt": "2019-05-09T19:49:24.201361Z",
- "updatedAt": "2019-05-09T19:49:24.201361Z",
- "connectionUrl": "jdbc:postgresql://db.example.com/mydb",
- "description": "My first source!"
}
Returns a source.
source required | string <= 1024 characters Example: my-source The name of the source. |
{- "type": "POSTGRESQL",
- "name": "my-source",
- "createdAt": "2019-05-09T19:49:24.201361Z",
- "updatedAt": "2019-05-09T19:49:24.201361Z",
- "connectionUrl": "jdbc:postgresql://db.example.com/mydb",
- "description": "My first source!"
}
Returns a list of sources.
limit | integer Default: 100 Example: limit=25 The number of results to return from offset. |
offset | integer Default: 0 The initial position from which to return results. |
{- "sources": [
- {
- "type": "POSTGRESQL",
- "name": "my-source",
- "createdAt": "2019-05-09T19:49:24.201361Z",
- "updatedAt": "2019-05-09T19:49:24.201361Z",
- "connectionUrl": "jdbc:postgresql://db.example.com/mydb",
- "description": "My first source!"
}
]
}
Creates a new dataset.
namespace required | string <= 1024 characters Example: my-namespace The name of the namespace. |
dataset required | string <= 1024 characters Example: my-dataset The name of the dataset. |
type required | string Value: "DB_TABLE" The type of the dataset. |
physicalName required | string The physical name of the table. |
sourceName required | string The name of the source associated with the table. |
required | Array of objects[ items ] The fields of the table. |
tags | Array of strings List of tags. |
description | string The description of the table. |
runId | string The ID associated with the run modifying the table. |
{- "type": "DB_TABLE",
- "physicalName": "public.mytable",
- "sourceName": "my-source",
- "fields": [
- {
- "name": "a",
- "type": "INTEGER"
}, - {
- "name": "b",
- "type": "TIMESTAMP"
}, - {
- "name": "c",
- "type": "INTEGER"
}, - {
- "name": "d",
- "type": "INTEGER"
}
], - "description": "My first dataset!"
}
{- "id": {
- "namespace": "my-namespace",
- "name": "my-dataset"
}, - "type": "DB_TABLE",
- "name": "my-dataset",
- "physicalName": "public.mytable",
- "createdAt": "2019-05-09T19:49:24.201361Z",
- "upodatedAt": "2019-05-09T19:49:24.201361Z",
- "namespace": "my-namespace",
- "sourceName": "my-source",
- "fields": [
- {
- "name'": "a",
- "type": "INTEGER",
- "tags": [ ]
}, - {
- "name'": "b",
- "type": "TIMESTAMP",
- "tags": [ ]
}, - {
- "name'": "c",
- "type": "INTEGER",
- "tags": [ ]
}, - {
- "name'": "d",
- "type": "INTEGER",
- "tags": [ ]
}
], - "tags": [ ],
- "lastModifiedAt": null,
- "description": "My first dataset!",
- "facets": { },
- "currentVersion": "b1d626a2-6d3a-475e-9ecf-943176d4a8c6"
}
Returns a dataset.
namespace required | string <= 1024 characters Example: my-namespace The name of the namespace. |
dataset required | string <= 1024 characters Example: my-dataset The name of the dataset. |
{- "id": {
- "namespace": "my-namespace",
- "name": "my-dataset"
}, - "type": "DB_TABLE",
- "name": "my-dataset",
- "physicalName": "public.mytable",
- "createdAt": "2019-05-09T19:49:24.201361Z",
- "upodatedAt": "2019-05-09T19:49:24.201361Z",
- "namespace": "my-namespace",
- "sourceName": "my-source",
- "fields": [
- {
- "name'": "a",
- "type": "INTEGER",
- "tags": [ ]
}, - {
- "name'": "b",
- "type": "TIMESTAMP",
- "tags": [ ]
}, - {
- "name'": "c",
- "type": "INTEGER",
- "tags": [ ]
}, - {
- "name'": "d",
- "type": "INTEGER",
- "tags": [ ]
}
], - "tags": [ ],
- "lastModifiedAt": null,
- "description": "My first dataset!",
- "facets": { },
- "currentVersion": "b1d626a2-6d3a-475e-9ecf-943176d4a8c6"
}
Soft deletes dataset. It will be un-deleted if new OpenLineage event containing this dataset comes.
namespace required | string <= 1024 characters Example: my-namespace The name of the namespace. |
dataset required | string <= 1024 characters Example: my-dataset The name of the dataset. |
{- "id": {
- "namespace": "my-namespace",
- "name": "my-dataset"
}, - "type": "DB_TABLE",
- "name": "my-dataset",
- "physicalName": "public.mytable",
- "createdAt": "2019-05-09T19:49:24.201361Z",
- "upodatedAt": "2019-05-09T19:49:24.201361Z",
- "namespace": "my-namespace",
- "sourceName": "my-source",
- "fields": [
- {
- "name'": "a",
- "type": "INTEGER",
- "tags": [ ]
}, - {
- "name'": "b",
- "type": "TIMESTAMP",
- "tags": [ ]
}, - {
- "name'": "c",
- "type": "INTEGER",
- "tags": [ ]
}, - {
- "name'": "d",
- "type": "INTEGER",
- "tags": [ ]
}
], - "tags": [ ],
- "lastModifiedAt": null,
- "description": "My first dataset!",
- "facets": { },
- "currentVersion": "b1d626a2-6d3a-475e-9ecf-943176d4a8c6"
}
Returns a version for a dataset.
namespace required | string <= 1024 characters Example: my-namespace The name of the namespace. |
dataset required | string <= 1024 characters Example: my-dataset The name of the dataset. |
version required | string <uuid> Example: ea9badc5-7cb2-49af-9a9f-155771d3a797 The ID of the job or dataset version. |
{- "id": {
- "namespace": "my-namespace",
- "name": "my-dataset"
}, - "type": "DB_TABLE",
- "name": "my-dataset",
- "physicalName": "public.mytable",
- "createdAt": "2019-05-09T19:49:24.201361Z",
- "version": "d224dac0-35d7-4d9b-bbbe-6fff1a8485ad",
- "namespace": "my-namespace",
- "sourceName": "my-source",
- "fields": [
- {
- "name": "a",
- "type": "INTEGER",
- "tags": [ ]
}, - {
- "name": "b",
- "type": "TIMESTAMP",
- "tags": [ ]
}, - {
- "name": "c",
- "type": "INTEGER",
- "tags": [ ]
}, - {
- "name": "d",
- "type": "INTEGER",
- "tags": [ ]
}
], - "tags": [ ],
- "description": "My first dataset!",
- "createdByRun": {
- "id": "0e89b60a-d80d-42ae-a12b-bd79ae1fb576",
- "createdAt": "2019-05-09T18:52:35.620763Z",
- "updatedAt": "2019-05-09T18:53:41.461122Z",
- "nominalStartTime": null,
- "nominalEndTime": null,
- "state": "COMPLETED",
- "startedAt": "2021-02-05T18:53:33.245492Z",
- "endedAt": "2021-02-05T18:53:41.461122Z",
- "durationMs": 8216,
- "args": {
- "email": "me@example.com",
- "emailOnFailure": false,
- "emailOnRetry": true,
- "retries": 1
}, - "jobVersion": {
- "namespace": "my-namespace",
- "name": "my_processing_job",
- "version": "0ba5b13b-350e-40f3-ba3f-cc00a4e92d11"
}, - "inputVersions": [
- {
- "namespace": "my-namespace",
- "name": "an_input_dataset",
- "version": "d224dac0-35d7-4d9b-bbbe-6fff1a8485ad"
}
], - "outputVersions": [
- {
- "namespace": "my-namespace",
- "name": "my-dataset",
- "version": "16e04029-2035-37f1-8206-d57be374333f"
}
]
}
}
Returns a list of versions for a dataset.
namespace required | string <= 1024 characters Example: my-namespace The name of the namespace. |
dataset required | string <= 1024 characters Example: my-dataset The name of the dataset. |
limit | integer Default: 100 Example: limit=25 The number of results to return from offset. |
offset | integer Default: 0 The initial position from which to return results. |
{- "versions": [
- {
- "id": {
- "namespace": "my-namespace",
- "name": "my-dataset"
}, - "type": "DB_TABLE",
- "name": "my-dataset",
- "physicalName": "public.mytable",
- "createdAt": "2019-05-09T19:49:24.201361Z",
- "version": "d224dac0-35d7-4d9b-bbbe-6fff1a8485ad",
- "namespace": "my-namespace",
- "sourceName": "my-source",
- "fields": [
- {
- "name": "a",
- "type": "INTEGER",
- "tags": [ ]
}, - {
- "name": "b",
- "type": "TIMESTAMP",
- "tags": [ ]
}, - {
- "name": "c",
- "type": "INTEGER",
- "tags": [ ]
}, - {
- "name": "d",
- "type": "INTEGER",
- "tags": [ ]
}
], - "tags": [ ],
- "description": "My first dataset!",
- "createdByRun": {
- "id": "0e89b60a-d80d-42ae-a12b-bd79ae1fb576",
- "createdAt": "2019-05-09T18:52:35.620763Z",
- "updatedAt": "2019-05-09T18:53:41.461122Z",
- "nominalStartTime": null,
- "nominalEndTime": null,
- "state": "COMPLETED",
- "startedAt": "2021-02-05T18:53:33.245492Z",
- "endedAt": "2021-02-05T18:53:41.461122Z",
- "durationMs": 8216,
- "args": {
- "email": "me@example.com",
- "emailOnFailure": false,
- "emailOnRetry": true,
- "retries": 1
}, - "jobVersion": {
- "namespace": "my-namespace",
- "name": "my_processing_job",
- "version": "0ba5b13b-350e-40f3-ba3f-cc00a4e92d11"
}, - "inputVersions": [
- {
- "namespace": "my-namespace",
- "name": "an_input_dataset",
- "version": "d224dac0-35d7-4d9b-bbbe-6fff1a8485ad"
}
], - "outputVersions": [
- {
- "namespace": "my-namespace",
- "name": "my-dataset",
- "version": "16e04029-2035-37f1-8206-d57be374333f"
}
]
}
}
]
}
Returns a list of datasets.
namespace required | string <= 1024 characters Example: my-namespace The name of the namespace. |
dataset required | string <= 1024 characters Example: my-dataset The name of the dataset. |
limit | integer Default: 100 Example: limit=25 The number of results to return from offset. |
offset | integer Default: 0 The initial position from which to return results. |
{- "datasets": [
- {
- "id": {
- "namespace": "my-namespace",
- "name": "my-dataset"
}, - "type": "DB_TABLE",
- "name": "my-dataset",
- "physicalName": "public.mytable",
- "createdAt": "2019-05-09T19:49:24.201361Z",
- "upodatedAt": "2019-05-09T19:49:24.201361Z",
- "namespace": "my-namespace",
- "sourceName": "my-source",
- "fields": [
- {
- "name'": "a",
- "type": "INTEGER",
- "tags": [ ]
}, - {
- "name'": "b",
- "type": "TIMESTAMP",
- "tags": [ ]
}, - {
- "name'": "c",
- "type": "INTEGER",
- "tags": [ ]
}, - {
- "name'": "d",
- "type": "INTEGER",
- "tags": [ ]
}
], - "tags": [ ],
- "lastModifiedAt": null,
- "description": "My first dataset!",
- "facets": { },
- "currentVersion": "b1d626a2-6d3a-475e-9ecf-943176d4a8c6"
}
], - "totalCount": 0
}
Tag an existing dataset.
namespace required | string <= 1024 characters Example: my-namespace The name of the namespace. |
dataset required | string <= 1024 characters Example: my-dataset The name of the dataset. |
tag required | string Example: SENSITIVE The name of the tag. |
{- "id": {
- "namespace": "my-namespace",
- "name": "my-dataset"
}, - "type": "DB_TABLE",
- "name": "my-dataset",
- "physicalName": "public.mytable",
- "createdAt": "2019-05-09T19:49:24.201361Z",
- "upodatedAt": "2019-05-09T19:49:24.201361Z",
- "namespace": "my-namespace",
- "sourceName": "my-source",
- "fields": [
- {
- "name'": "a",
- "type": "INTEGER",
- "tags": [ ]
}, - {
- "name'": "b",
- "type": "TIMESTAMP",
- "tags": [ ]
}, - {
- "name'": "c",
- "type": "INTEGER",
- "tags": [ ]
}, - {
- "name'": "d",
- "type": "INTEGER",
- "tags": [ ]
}
], - "tags": [ ],
- "lastModifiedAt": null,
- "description": "My first dataset!",
- "facets": { },
- "currentVersion": "b1d626a2-6d3a-475e-9ecf-943176d4a8c6"
}
Tag an existing field of a dataset.
namespace required | string <= 1024 characters Example: my-namespace The name of the namespace. |
dataset required | string <= 1024 characters Example: my-dataset The name of the dataset. |
field required | string Example: my_field The name of the field. |
tag required | string Example: SENSITIVE The name of the tag. |
{- "id": {
- "namespace": "my-namespace",
- "name": "my-dataset"
}, - "type": "DB_TABLE",
- "name": "my-dataset",
- "physicalName": "public.mytable",
- "createdAt": "2019-05-09T19:49:24.201361Z",
- "upodatedAt": "2019-05-09T19:49:24.201361Z",
- "namespace": "my-namespace",
- "sourceName": "my-source",
- "fields": [
- {
- "name'": "a",
- "type": "INTEGER",
- "tags": [ ]
}, - {
- "name'": "b",
- "type": "TIMESTAMP",
- "tags": [ ]
}, - {
- "name'": "c",
- "type": "INTEGER",
- "tags": [ ]
}, - {
- "name'": "d",
- "type": "INTEGER",
- "tags": [ ]
}
], - "tags": [ ],
- "lastModifiedAt": null,
- "description": "My first dataset!",
- "facets": { },
- "currentVersion": "b1d626a2-6d3a-475e-9ecf-943176d4a8c6"
}
Creates a new job object. All job objects are immutable and are uniquely identified by a generated ID. Marquez will create a version of a job each time the contents of the object is modified. For example, the location
of a job may change over time resulting in new versions. The accumulated versions can be listed, used to rerun a specific job version or possibly help debug a failed job run.
namespace required | string <= 1024 characters Example: my-namespace The name of the namespace. |
job required | string <= 1024 characters Example: my-job The name of the job. |
object The ID of the job. | |
type required | string (JobType) Enum: "BATCH" "STREAM" "SERVICE" The type of the job. |
required | Array of objects (DatasetId) unique [ items ] The set of input datasets. |
required | Array of objects (DatasetId) unique [ items ] The set of output datasets. |
location | string <URL> The URL of the job source code or artifact. |
context | object Deprecated A key/value pair that must be of type |
description | string The description of the job. |
runId | string An optional run ID used to associate a job version to an existing job run. |
{- "type": "BATCH",
- "inputs": [
- {
- "namespace": "my-namespace",
- "name": "my-dataset"
}
], - "outputs": [ ],
- "context": {
- "sql": "SELECT * FROM mytable;"
}, - "description": "My first job!"
}
{- "id": {
- "namespace": "my-namespace",
- "name": "my-job"
}, - "type": "BATCH",
- "name": "my-job",
- "createdAt": "2019-05-09T19:49:24.201361Z",
- "updatedAt": "2019-05-09T19:49:24.201361Z",
- "namespace": "my-namespace",
- "inputs": [
- {
- "namespace": "my-namespace",
- "name": "my-dataset"
}
], - "outputs": [ ],
- "context": {
- "SQL": "SELECT * FROM mytable;"
}, - "description": "My first job!",
- "latestRun": null,
- "facets": { },
- "currentVersion": "b1d626a2-6d3a-475e-9ecf-943176d4a8c6"
}
Retrieve a job.
namespace required | string <= 1024 characters Example: my-namespace The name of the namespace. |
job required | string <= 1024 characters Example: my-job The name of the job. |
{- "id": {
- "namespace": "my-namespace",
- "name": "my-job"
}, - "type": "BATCH",
- "name": "my-job",
- "createdAt": "2019-05-09T19:49:24.201361Z",
- "updatedAt": "2019-05-09T19:49:24.201361Z",
- "namespace": "my-namespace",
- "inputs": [
- {
- "namespace": "my-namespace",
- "name": "my-dataset"
}
], - "outputs": [ ],
- "context": {
- "SQL": "SELECT * FROM mytable;"
}, - "description": "My first job!",
- "latestRun": null,
- "facets": { },
- "currentVersion": "b1d626a2-6d3a-475e-9ecf-943176d4a8c6"
}
Soft deletes job. It will be un-deleted if new OpenLineage event containing this job comes.
namespace required | string <= 1024 characters Example: my-namespace The name of the namespace. |
job required | string <= 1024 characters Example: my-job The name of the job. |
{- "id": {
- "namespace": "my-namespace",
- "name": "my-job"
}, - "type": "BATCH",
- "name": "my-job",
- "createdAt": "2019-05-09T19:49:24.201361Z",
- "updatedAt": "2019-05-09T19:49:24.201361Z",
- "namespace": "my-namespace",
- "inputs": [
- {
- "namespace": "my-namespace",
- "name": "my-dataset"
}
], - "outputs": [ ],
- "context": {
- "SQL": "SELECT * FROM mytable;"
}, - "description": "My first job!",
- "latestRun": null,
- "facets": { },
- "currentVersion": "b1d626a2-6d3a-475e-9ecf-943176d4a8c6"
}
Returns a list of jobs.
namespace required | string <= 1024 characters Example: my-namespace The name of the namespace. |
limit | integer Default: 100 Example: limit=25 The number of results to return from offset. |
offset | integer Default: 0 The initial position from which to return results. |
{- "jobs": [
- {
- "id": {
- "namespace": "my-namespace",
- "name": "my-job"
}, - "type": "BATCH",
- "name": "my-job",
- "createdAt": "2019-05-09T19:49:24.201361Z",
- "updatedAt": "2019-05-09T19:49:24.201361Z",
- "namespace": "my-namespace",
- "inputs": [
- {
- "namespace": "my-namespace",
- "name": "my-dataset"
}
], - "outputs": [ ],
- "context": {
- "SQL": "SELECT * FROM mytable;"
}, - "description": "My first job!",
- "latestRun": null,
- "facets": { },
- "currentVersion": "b1d626a2-6d3a-475e-9ecf-943176d4a8c6"
}
], - "totalCount": 0
}
Returns a version for a job.
namespace required | string <= 1024 characters Example: my-namespace The name of the namespace. |
job required | string <= 1024 characters Example: my-job The name of the job. |
version required | string <uuid> Example: ea9badc5-7cb2-49af-9a9f-155771d3a797 The ID of the job or dataset version. |
{- "id": {
- "namespace": "my-namespace",
- "name": "my-job",
- "version": "56472c57-a2ef-4218-b7b7-d2af02a343fd"
}, - "type": "BATCH",
- "name": "my-job",
- "version": "56472c57-a2ef-4218-b7b7-d2af02a343fd",
- "createdAt": "2019-05-09T19:49:24.201361Z",
- "namespace": "my-namespace",
- "inputs": [
- {
- "namespace": "my-namespace",
- "name": "my-dataset"
}
], - "outputs": [ ],
- "context": {
- "SQL": "SELECT * FROM mytable;"
}, - "description": "My first job!",
- "facets": { }
}
Returns a list of versions for a job.
namespace required | string <= 1024 characters Example: my-namespace The name of the namespace. |
job required | string <= 1024 characters Example: my-job The name of the job. |
{- "versions": [
- {
- "id": {
- "namespace": "my-namespace",
- "name": "my-job",
- "version": "56472c57-a2ef-4218-b7b7-d2af02a343fd"
}, - "type": "BATCH",
- "name": "my-job",
- "version": "56472c57-a2ef-4218-b7b7-d2af02a343fd",
- "createdAt": "2019-05-09T19:49:24.201361Z",
- "namespace": "my-namespace",
- "inputs": [
- {
- "namespace": "my-namespace",
- "name": "my-dataset"
}
], - "outputs": [ ],
- "context": {
- "SQL": "SELECT * FROM mytable;"
}, - "description": "My first job!",
- "facets": { }
}
]
}
Creates a new run object for a job.
namespace required | string <= 1024 characters Example: my-namespace The name of the namespace. |
job required | string <= 1024 characters Example: my-job The name of the job. |
id | string <uuid> An optional user-provided unique ID of the run. A run ID must be an UUID. If an ID for the run is not provided, a random UUID will be generated for the given run. |
nominalStartTime | string <date-time> An ISO-8601 timestamp representing the nominal start time of the run. |
nominalEndTime | string <date-time> An ISO-8601 timestamp representing the nominal end time of the run. |
args | object The arguments of the run. |
{- "args": {
- "email": "me@example.com",
- "emailOnFailure": false,
- "emailOnRetry": true,
- "retries": 1
}
}
{- "id": "870492da-ecfb-4be0-91b9-9a89ddd3db90",
- "createdAt": "2019-05-09T19:49:24.201361Z",
- "updatedAt": "2019-05-09T19:49:24.201361Z",
- "nominalStartTime": null,
- "nominalEndTime": null,
- "state": "RUNNING",
- "startedAt": "2019-05-09T15:17:32.690346",
- "endedAt": null,
- "durationMs": null,
- "args": {
- "email": "me@example.com",
- "emailOnFailure": "false",
- "emailOnRetry": "true",
- "retries": "1"
}, - "facets": { }
}
Returns a list of runs for a job.
namespace required | string <= 1024 characters Example: my-namespace The name of the namespace. |
job required | string <= 1024 characters Example: my-job The name of the job. |
limit | integer Default: 100 Example: limit=25 The number of results to return from offset. |
offset | integer Default: 0 The initial position from which to return results. |
{- "runs": [
- {
- "id": "870492da-ecfb-4be0-91b9-9a89ddd3db90",
- "createdAt": "2019-05-09T19:49:24.201361Z",
- "updatedAt": "2019-05-09T19:49:24.201361Z",
- "nominalStartTime": null,
- "nominalEndTime": null,
- "state": "COMPLETED",
- "startedAt": "2019-05-09T15:17:32.690346",
- "endedAt": "2019-05-09T20:05:46.815920Z",
- "durationMs": 4250894125,
- "args": {
- "email": "me@example.com",
- "emailOnFailure": "false",
- "emailOnRetry": "true",
- "retries": "1"
}, - "context": {
- "SQL": "SELECT * FROM mytable;"
}, - "facets": { }
}
]
}
Retrieve a run.
id required | string <uuid> Example: ea9badc5-7cb2-49af-9a9f-155771d3a797 The ID of the run. |
{- "id": "870492da-ecfb-4be0-91b9-9a89ddd3db90",
- "createdAt": "2019-05-09T19:49:24.201361Z",
- "updatedAt": "2019-05-09T19:49:24.201361Z",
- "nominalStartTime": null,
- "nominalEndTime": null,
- "state": "RUNNING",
- "startedAt": "2019-05-09T15:17:32.690346",
- "endedAt": null,
- "durationMs": null,
- "args": {
- "email": "me@example.com",
- "emailOnFailure": "false",
- "emailOnRetry": "true",
- "retries": "1"
}, - "facets": { }
}
Retrieve run or job facets for a run.
id required | string <uuid> Example: ea9badc5-7cb2-49af-9a9f-155771d3a797 The ID of the run. |
type required | string Enum: "run" "job" Indicates if should return job or run facets. |
{- "runId": "string",
- "facets": {
}
}
Marks the run as RUNNING
.
id required | string <uuid> Example: ea9badc5-7cb2-49af-9a9f-155771d3a797 The ID of the run. |
at | string <date-time> An ISO-8601 timestamp representing the time when the run transitioned. |
{- "id": "870492da-ecfb-4be0-91b9-9a89ddd3db90",
- "createdAt": "2019-05-09T19:49:24.201361Z",
- "updatedAt": "2019-05-09T19:49:24.201361Z",
- "nominalStartTime": null,
- "nominalEndTime": null,
- "state": "RUNNING",
- "startedAt": "2019-05-09T15:17:32.690346",
- "endedAt": null,
- "durationMs": null,
- "args": {
- "email": "me@example.com",
- "emailOnFailure": "false",
- "emailOnRetry": "true",
- "retries": "1"
}, - "facets": { }
}
Marks the run as COMPLETED
.
id required | string <uuid> Example: ea9badc5-7cb2-49af-9a9f-155771d3a797 The ID of the run. |
at | string <date-time> An ISO-8601 timestamp representing the time when the run transitioned. |
{- "id": "870492da-ecfb-4be0-91b9-9a89ddd3db90",
- "createdAt": "2019-05-09T19:49:24.201361Z",
- "updatedAt": "2019-05-09T19:49:24.201361Z",
- "nominalStartTime": null,
- "nominalEndTime": null,
- "state": "RUNNING",
- "startedAt": "2019-05-09T15:17:32.690346",
- "endedAt": null,
- "durationMs": null,
- "args": {
- "email": "me@example.com",
- "emailOnFailure": "false",
- "emailOnRetry": "true",
- "retries": "1"
}, - "facets": { }
}
Marks the run as FAILED
.
id required | string <uuid> Example: ea9badc5-7cb2-49af-9a9f-155771d3a797 The ID of the run. |
at | string <date-time> An ISO-8601 timestamp representing the time when the run transitioned. |
{- "id": "870492da-ecfb-4be0-91b9-9a89ddd3db90",
- "createdAt": "2019-05-09T19:49:24.201361Z",
- "updatedAt": "2019-05-09T19:49:24.201361Z",
- "nominalStartTime": null,
- "nominalEndTime": null,
- "state": "RUNNING",
- "startedAt": "2019-05-09T15:17:32.690346",
- "endedAt": null,
- "durationMs": null,
- "args": {
- "email": "me@example.com",
- "emailOnFailure": "false",
- "emailOnRetry": "true",
- "retries": "1"
}, - "facets": { }
}
Marks the run as ABORTED
.
id required | string <uuid> Example: ea9badc5-7cb2-49af-9a9f-155771d3a797 The ID of the run. |
at | string <date-time> An ISO-8601 timestamp representing the time when the run transitioned. |
{- "id": "870492da-ecfb-4be0-91b9-9a89ddd3db90",
- "createdAt": "2019-05-09T19:49:24.201361Z",
- "updatedAt": "2019-05-09T19:49:24.201361Z",
- "nominalStartTime": null,
- "nominalEndTime": null,
- "state": "RUNNING",
- "startedAt": "2019-05-09T15:17:32.690346",
- "endedAt": null,
- "durationMs": null,
- "args": {
- "email": "me@example.com",
- "emailOnFailure": "false",
- "emailOnRetry": "true",
- "retries": "1"
}, - "facets": { }
}
Receive, process, and store lineage metadata using the OpenLineage standard.
{- "eventType": "COMPLETE",
- "eventTime": "2019-05-09T19:49:24.201361Z",
- "run": {
- "runId": "d46e465b-d358-4d32-83d4-df660ff614dd"
}, - "job": {
- "namespace": "my-namespace",
- "name": "my-job"
}, - "outputs": [
- {
- "namespace": "my-namespace",
- "name": "my-output",
- "facets": {
- "schema": {
- "fields": [
- {
- "name": "a",
- "type": "INTEGER"
}, - {
- "name": "b",
- "type": "TIMESTAMP"
}, - {
- "name": "c",
- "type": "INTEGER"
}, - {
- "name": "d",
- "type": "INTEGER"
}
]
}
}
}
],
}
nodeId required | string Example: nodeId=dataset:food_delivery:public.delivery_7_days The ID of the node. A node can either be a dataset node, a dataset field node or a job node. The format of nodeId for dataset is |
depth | integer Default: 20 Depth of lineage graph to create. |
{- "graph": [
- {
- "id": "string",
- "type": "JOB",
- "data": {
- "id": {
- "namespace": "my-namespace",
- "name": "my-dataset"
}, - "type": "DB_TABLE",
- "name": "my-dataset",
- "physicalName": "public.mytable",
- "createdAt": "2019-05-09T19:49:24.201361Z",
- "upodatedAt": "2019-05-09T19:49:24.201361Z",
- "namespace": "my-namespace",
- "sourceName": "my-source",
- "fields": [
- {
- "name'": "a",
- "type": "INTEGER",
- "tags": [ ]
}, - {
- "name'": "b",
- "type": "TIMESTAMP",
- "tags": [ ]
}, - {
- "name'": "c",
- "type": "INTEGER",
- "tags": [ ]
}, - {
- "name'": "d",
- "type": "INTEGER",
- "tags": [ ]
}
], - "tags": [ ],
- "lastModifiedAt": null,
- "description": "My first dataset!",
- "facets": { },
- "currentVersion": "b1d626a2-6d3a-475e-9ecf-943176d4a8c6"
}, - "inEdges": [
- {
- "origin": "string",
- "destination": "string"
}
], - "outEdges": [
- {
- "origin": "string",
- "destination": "string"
}
]
}
]
}
nodeId required | string Example: nodeId=dataset:food_delivery:public.delivery_7_days The ID of the node. A node can either be a dataset node, a dataset field node or a job node. The format of nodeId for dataset is |
depth | integer Default: 20 Depth of lineage graph to create. |
withDownstream | boolean Default: false Determines if downstream lineage should be returned. |
{- "graph": [
- {
- "id": "string",
- "type": "JOB",
- "data": {
- "id": {
- "namespace": "my-namespace",
- "name": "my-dataset"
}, - "type": "DB_TABLE",
- "name": "my-dataset",
- "physicalName": "public.mytable",
- "createdAt": "2019-05-09T19:49:24.201361Z",
- "upodatedAt": "2019-05-09T19:49:24.201361Z",
- "namespace": "my-namespace",
- "sourceName": "my-source",
- "fields": [
- {
- "name'": "a",
- "type": "INTEGER",
- "tags": [ ]
}, - {
- "name'": "b",
- "type": "TIMESTAMP",
- "tags": [ ]
}, - {
- "name'": "c",
- "type": "INTEGER",
- "tags": [ ]
}, - {
- "name'": "d",
- "type": "INTEGER",
- "tags": [ ]
}
], - "tags": [ ],
- "lastModifiedAt": null,
- "description": "My first dataset!",
- "facets": { },
- "currentVersion": "b1d626a2-6d3a-475e-9ecf-943176d4a8c6"
}, - "inEdges": [
- {
- "origin": "string",
- "destination": "string"
}
], - "outEdges": [
- {
- "origin": "string",
- "destination": "string"
}
]
}
]
}
Creates a new tag object.
tag required | string Example: SENSITIVE The name of the tag. |
description | string The description of the tag. |
{- "description": "My first tag!"
}
{- "tags": [
- {
- "name": "SENSITIVE",
- "description": "Contains sensitive information"
}
]
}
Returns a list of tags.
limit | integer Default: 100 Example: limit=25 The number of results to return from offset. |
offset | integer Default: 0 The initial position from which to return results. |
{- "tags": [
- {
- "name": "SENSITIVE",
- "description": "Contains sensitive information"
}
]
}
Returns one or more datasets and jobs of your query.
q required | string Example: q=my-dataset Query containing pattern to match; datasets and jobs pattern matching is string based and case-insensitive. Use percent sign ( |
filter | string Example: filter=dataset Filters the results of your query by |
sort | string Example: sort=name Sorts the results of your query by |
limit | integer Default: 100 Example: limit=25 The number of results to return from offset. |
namespace | string <= 1024 characters Example: namespace=my-namespace Match jobs or datasets within the given namespace. |
before | stringYYYY-MM-DD Example: before=2022-09-15 Match jobs or datasets before |
after | stringYYYY-MM-DD Example: after=2022-09-15 Match jobs or datasets after |
{- "totalCount": 1,
- "results": [
- {
- "type": "DATASET",
- "name": "public.delivery_7_days",
- "updatedAt": "2019-05-09T19:49:24.201361Z",
- "namespace": "food_delivery",
- "nodeId": "dataset:food_delivery:public.delivery_7_days"
}
]
}