Skip to the content.

Lifecycle

Marquez captures the runs of a job, and changes as they happen.

Run states:

NEW -(start)-> RUNNING -(complete)-> COMPLETED
                       -(abort)-> ABORTED
                       -(fail)-> FAILED

Static inspection of job lineage

When the job lineage can be inspected from its static definition the metadata is captured as follows:

When the run of the job starts

When the run of the job ends

Dynamic inspection of job lineage

If the job can not be introspected statically, we will have to capture the information as the job is running. Then all the updates will be sent when the job is actually completing. (both steps above are executed when the job finishes)

When the run of the job starts

When the run of the job ends

Unified case

When the run of the job starts

extract info from the job

When the run of the job ends