Beam support

[jbadeau]

Question

As Apache Beam can run pipelines on Spark. Will Spline agents work if the beam pipelines are run on Spark?

[wajda]

I don't know how Beam works on Spark under the hood, but here's some thoughts:

The Spline agent captures DataFrame level transformations by listening to Spark query execution events emitted from write commands.
So in theory, as long as Beam utilizes DataFrame's and calls one of the supported write commands then it should be supported out of the box.

If Beam implements a custom write command, but still utilizes DataFrame API then a plugin will be required to capture the lineage.

The worst case would be if Beam works on a RDD level, that is not supported at the moment (see #33).