Not showing data Lineage for multiple RUNNING jobs until they are marked COMPLETED

[kspmmitt]

Hello,

I have multiple jobs taking/giving data on common input/output data sets and also these data sets/jobs placed in same namespace. But if job RUNs in sequence are not marked COMPLETED then it doesn't show any details like data facets and also shows jobs as individual nodes i.e. without connected to each other in data lineage fashion.

Also, same thing is happing for a job itself where dataset and job nodes are not showing connected and dataset not showing any facet.

Marquez version - 0.51.1

Can you please help?

Rgds,
Kamal

[boring-cyborg]

Thanks for opening your first issue in the Marquez project! Please be sure to follow the issue template!

[kspmmitt]

I think issue is due to run life cycle state. I am emitting events by OpenLineage and as per page link below, looks like RUNNING state will not emit any dataset and that causes for multiple or with in a job connection.

But then RUNNING state is meaningless, also for streaming jobs how it will be taken care, by emitting COMPLETE in between and START again? Otherwise, subsequent jobs will not be connected. That looks bit strange but please confirm the behavior if that is the case.

[https://openlineage.io/docs/spec/run-cycle]