Deployment Overview

Helm Chart

Marquez uses Helm to manage deployments onto Kubernetes in a cloud environment. The chart and templates for the HTTP API server and Web UI are maintained in the Marquez repository and can be found in the chart directory. The chart’s base values.yaml file includes an option to easily override deployment settings.

Note: The Marquez HTTP API server and Web UI images are publshed to DockerHub.

Database

The Marquez HTTP API server relies only on PostgreSQL to store dataset, job, and run metadata allowing for minimal operational overhead. We recommend a cloud provided databases, such as AWS RDS, when deploying Marquez onto Kubernetes.

Architecture

DOCKER

Figure 1: Minimal Marquez deployment via Docker.

KUBERNETES

Figure 2: Marquez deployment via Kubernetes.

COMPONENTS

Component Image Description
Marquez Web UI marquezproject/marquez-web The web UI used to view metadata.
Marquez HTTP API marquezproject/marquez The core API used to collect metadata using OpenLineage.
Database bitnami/postgresql or cloud provided A PostgreSQL instance used to store metadata.
Workflow User-provided A workflow using an OpenLineage integration to send lineage metadata to Marquez.

Authentication

Our clients support authentication by automatically sending an API key on each request via Bearer Auth when configured on client instantiation. By default, the Marquez HTTP API does not require any form of authentication or authorization.

Next Steps

The following guides will help you and your team effectively deploy and manage Marquez in a cloud environment: