Skip to content

[Spark] Add support for OpenLineage for both batch and stream jobs #168

@dmetasoul01

Description

@dmetasoul01
  1. Parse input relation as input datasets
  2. Extract lakesoul's table name during commit as output dataset
  3. Extract spark's job name (app name) as job
  4. Emit run event for every commit

Metadata

Metadata

Labels

enhancementNew feature or requestsparkspark support into lakesoul

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions