Marquez (0.34.0)

Download OpenAPI specification:Download

Marquez is an open source metadata service for the collection, aggregation, and visualization of a data ecosystem's metadata.

Namespaces

Create a namespace

Creates a new namespace object. A namespace enables the contextual grouping of related jobs and datasets. Namespaces must contain only letters (a-z, A-Z), numbers (0-9), underscores (_), dashes (-), colons (:), slashes (/), or dots (.). A namespace is case-insensitive with a maximum length of 1024 characters. Note jobs and datasets will be unique within a namespace, but not across namespaces.

path Parameters

namespace

required

string <= 1024 characters

Example: my-namespace

The name of the namespace.

Request Body schema: application/json

ownerName required	string The owner of the namespace.
description	string The description of the namespace.

Responses

Request samples

Payload

Content type

application/json

{"ownerName": "me",
"description": "My first namespace!"
}

Response samples

200

Content type

application/json

{"name": "my-namespace",
"createdAt": "2019-05-09T19:49:24.201361Z",
"updatedAt": "2019-05-09T19:49:24.201361Z",
"ownerName": "me",
"description": "My first namespace!"
}

Retrieve a namespace

Returns a namespace.

path Parameters

namespace

required

string <= 1024 characters

Example: my-namespace

The name of the namespace.

Responses

Response samples

200

Content type

application/json

{"name": "my-namespace",
"createdAt": "2019-05-09T19:49:24.201361Z",
"updatedAt": "2019-05-09T19:49:24.201361Z",
"ownerName": "me",
"description": "My first namespace!"
}

Deletes a namespace

Soft deletes a namespace, and every job and dataset inside. On next event containing this namespace, the namespace will be undeleted.

path Parameters

namespace

required

string <= 1024 characters

Example: my-namespace

The name of the namespace.

Responses

Response samples

200

Content type

application/json

{"name": "my-namespace",
"createdAt": "2019-05-09T19:49:24.201361Z",
"updatedAt": "2019-05-09T19:49:24.201361Z",
"ownerName": "me",
"description": "My first namespace!"
}

List all namespaces

Returns a list of namespaces.

query Parameters

limit	integer Default: 100 Example: limit=25 The number of results to return from offset
offset	integer Default: 0 The initial position from which to return results

Responses

Response samples

200

Content type

application/json

{"namespaces": [{"name": "my-namespace",
"createdAt": "2019-05-09T19:49:24.201361Z",
"updatedAt": "2019-05-09T19:49:24.201361Z",
"ownerName": "me",
"description": "My first namespace!"
}
]
}

Events

List all received OpenLineage events.

Returns a list of OpenLineage events, sorted in direction of passed sort parameter. By default it is desc.

query Parameters

sortDirection	string Example: sortDirection=name Sorts the results of your query by indicated direction `asc` or `desc`.
before	string <date-time> Example: before=2022-09-15T07:47:19Z Returns events before passed date.
after	string <date-time> Example: after=2022-09-15T07:47:19Z Returns events after passed date.
limit	integer Default: 100 Example: limit=25 The number of results to return from offset

Responses

Response samples

200

Content type

application/json

{"events": [{"eventType": "COMPLETE",
"eventTime": "2019-05-09T19:49:24.201361Z",
"run": {"runId": "d46e465b-d358-4d32-83d4-df660ff614dd"
},
"job": {"namespace": "my-namespace",
"name": "my-job"
},
"outputs": [{"namespace": "my-namespace",
"name": "my-output",
"facets": {"schema": {"_producer": "https://github.com/OpenLineage/OpenLineage/blob/v1-0-0/client",
"_schemaURL": "https://github.com/OpenLineage/OpenLineage/blob/v1-0-0/spec/OpenLineage.json#/definitions/SchemaDatasetFacet",
"fields": [{"name": "a",
"type": "INTEGER"
},
{"name": "b",
"type": "TIMESTAMP"
},
{"name": "c",
"type": "INTEGER"
},
{"name": "d",
"type": "INTEGER"
}
]
}
}
}
],
"producer": "https://github.com/OpenLineage/OpenLineage/blob/v1-0-0/client"
}
]
}

Sources

Create a source Deprecated

Creates a new source object. A source is the physical location of a dataset such as a table in PostgreSQL, or topic in Kafka. A source enables the grouping of physical datasets to their physical source.

path Parameters

source

required

string <= 1024 characters

Example: my-source

The name of the source.

Request Body schema: application/json

type required	string The type of the source.
connectionUrl required	string <URL> The URL to the location of the source.
description	string The description of the source.

Responses

Request samples

Payload

Content type

application/json

{"type": "POSTGRESQL",
"connectionUrl": "jdbc:postgresql://db.example.com/mydb",
"description": "My first source!"
}

Response samples

200

Content type

application/json

{"type": "POSTGRESQL",
"name": "my-source",
"createdAt": "2019-05-09T19:49:24.201361Z",
"updatedAt": "2019-05-09T19:49:24.201361Z",
"connectionUrl": "jdbc:postgresql://db.example.com/mydb",
"description": "My first source!"
}

Retrieve a source

Returns a source.

path Parameters

source

required

string <= 1024 characters

Example: my-source

The name of the source.

Responses

Response samples

200

Content type

application/json

{"type": "POSTGRESQL",
"name": "my-source",
"createdAt": "2019-05-09T19:49:24.201361Z",
"updatedAt": "2019-05-09T19:49:24.201361Z",
"connectionUrl": "jdbc:postgresql://db.example.com/mydb",
"description": "My first source!"
}

List all sources

Returns a list of sources.

query Parameters

limit	integer Default: 100 Example: limit=25 The number of results to return from offset
offset	integer Default: 0 The initial position from which to return results

Responses

Response samples

200

Content type

application/json

{"sources": [{"type": "POSTGRESQL",
"name": "my-source",
"createdAt": "2019-05-09T19:49:24.201361Z",
"updatedAt": "2019-05-09T19:49:24.201361Z",
"connectionUrl": "jdbc:postgresql://db.example.com/mydb",
"description": "My first source!"
}
]
}

Datasets

Create a dataset Deprecated

Creates a new dataset.

path Parameters

namespace required	string <= 1024 characters Example: my-namespace The name of the namespace.
dataset required	string <= 1024 characters Example: my-dataset The name of the dataset.

Request Body schema: application/json

Any of

type required	string Value: "DB_TABLE" The type of the dataset.
physicalName required	string The physical name of the table.
sourceName required	string The name of the source associated with the table.
required	Array of objects[ items ] The fields of the table.
tags	Array of strings List of tags.
description	string The description of the table.
runId	string The ID associated with the run modifying the table.

Responses

Request samples

Payload

Content type

application/json

Example

{"type": "DB_TABLE",
"physicalName": "public.mytable",
"sourceName": "my-source",
"fields": [{"name": "a",
"type": "INTEGER"
},
{"name": "b",
"type": "TIMESTAMP"
},
{"name": "c",
"type": "INTEGER"
},
{"name": "d",
"type": "INTEGER"
}
],
"description": "My first dataset!"
}

Response samples

200

Content type

application/json

{"id": {"namespace": "my-namespace",
"name": "my-dataset"
},
"type": "DB_TABLE",
"name": "my-dataset",
"physicalName": "public.mytable",
"createdAt": "2019-05-09T19:49:24.201361Z",
"upodatedAt": "2019-05-09T19:49:24.201361Z",
"namespace": "my-namespace",
"sourceName": "my-source",
"fields": [{"name'": "a",
"type": "INTEGER",
"tags": [ ]
},
{"name'": "b",
"type": "TIMESTAMP",
"tags": [ ]
},
{"name'": "c",
"type": "INTEGER",
"tags": [ ]
},
{"name'": "d",
"type": "INTEGER",
"tags": [ ]
}
],
"tags": [ ],
"lastModifiedAt": null,
"description": "My first dataset!",
"facets": { },
"currentVersion": "b1d626a2-6d3a-475e-9ecf-943176d4a8c6"
}

Retrieve a dataset

Returns a dataset.

path Parameters

namespace required	string <= 1024 characters Example: my-namespace The name of the namespace.
dataset required	string <= 1024 characters Example: my-dataset The name of the dataset.

Responses

Response samples

200

Content type

application/json

{"id": {"namespace": "my-namespace",
"name": "my-dataset"
},
"type": "DB_TABLE",
"name": "my-dataset",
"physicalName": "public.mytable",
"createdAt": "2019-05-09T19:49:24.201361Z",
"upodatedAt": "2019-05-09T19:49:24.201361Z",
"namespace": "my-namespace",
"sourceName": "my-source",
"fields": [{"name'": "a",
"type": "INTEGER",
"tags": [ ]
},
{"name'": "b",
"type": "TIMESTAMP",
"tags": [ ]
},
{"name'": "c",
"type": "INTEGER",
"tags": [ ]
},
{"name'": "d",
"type": "INTEGER",
"tags": [ ]
}
],
"tags": [ ],
"lastModifiedAt": null,
"description": "My first dataset!",
"facets": { },
"currentVersion": "b1d626a2-6d3a-475e-9ecf-943176d4a8c6"
}

Soft deletes dataset.

Soft deletes dataset. It will be un-deleted if new OpenLineage event containing this dataset comes.

path Parameters

namespace required	string <= 1024 characters Example: my-namespace The name of the namespace.
dataset required	string <= 1024 characters Example: my-dataset The name of the dataset.

Responses

Response samples

200

Content type

application/json

{"id": {"namespace": "my-namespace",
"name": "my-dataset"
},
"type": "DB_TABLE",
"name": "my-dataset",
"physicalName": "public.mytable",
"createdAt": "2019-05-09T19:49:24.201361Z",
"upodatedAt": "2019-05-09T19:49:24.201361Z",
"namespace": "my-namespace",
"sourceName": "my-source",
"fields": [{"name'": "a",
"type": "INTEGER",
"tags": [ ]
},
{"name'": "b",
"type": "TIMESTAMP",
"tags": [ ]
},
{"name'": "c",
"type": "INTEGER",
"tags": [ ]
},
{"name'": "d",
"type": "INTEGER",
"tags": [ ]
}
],
"tags": [ ],
"lastModifiedAt": null,
"description": "My first dataset!",
"facets": { },
"currentVersion": "b1d626a2-6d3a-475e-9ecf-943176d4a8c6"
}

Retrieve a version for a dataset

Returns a version for a dataset.

path Parameters

namespace required	string <= 1024 characters Example: my-namespace The name of the namespace.
dataset required	string <= 1024 characters Example: my-dataset The name of the dataset.
version required	string <uuid> Example: ea9badc5-7cb2-49af-9a9f-155771d3a797 The ID of the job or dataset version.

Responses

Response samples

200

Content type

application/json

{"id": {"namespace": "my-namespace",
"name": "my-dataset"
},
"type": "DB_TABLE",
"name": "my-dataset",
"physicalName": "public.mytable",
"createdAt": "2019-05-09T19:49:24.201361Z",
"version": "d224dac0-35d7-4d9b-bbbe-6fff1a8485ad",
"namespace": "my-namespace",
"sourceName": "my-source",
"fields": [{"name": "a",
"type": "INTEGER",
"tags": [ ]
},
{"name": "b",
"type": "TIMESTAMP",
"tags": [ ]
},
{"name": "c",
"type": "INTEGER",
"tags": [ ]
},
{"name": "d",
"type": "INTEGER",
"tags": [ ]
}
],
"tags": [ ],
"description": "My first dataset!",
"createdByRun": {"id": "0e89b60a-d80d-42ae-a12b-bd79ae1fb576",
"createdAt": "2019-05-09T18:52:35.620763Z",
"updatedAt": "2019-05-09T18:53:41.461122Z",
"nominalStartTime": null,
"nominalEndTime": null,
"state": "COMPLETED",
"startedAt": "2021-02-05T18:53:33.245492Z",
"endedAt": "2021-02-05T18:53:41.461122Z",
"durationMs": 8216,
"args": {"email": "me@example.com",
"emailOnFailure": false,
"emailOnRetry": true,
"retries": 1
},
"jobVersion": {"namespace": "my-namespace",
"name": "my_processing_job",
"version": "0ba5b13b-350e-40f3-ba3f-cc00a4e92d11"
},
"inputVersions": [{"namespace": "my-namespace",
"name": "an_input_dataset",
"version": "d224dac0-35d7-4d9b-bbbe-6fff1a8485ad"
}
],
"outputVersions": [{"namespace": "my-namespace",
"name": "my-dataset",
"version": "16e04029-2035-37f1-8206-d57be374333f"
}