Visualize from hdfs?

[maybenotconnor]

Hi,
I’m setting up spline in an environment with the limitation of being unable to use the API. I’m currently saving the lineage with the hdfs dispatcher, but is there any way to interpret it with the ui? It doesn’t seem that it can be directed to a file.

I know this isn’t the intended usage though I would really like to try spline for use in debugging. Just need a way to interpret the data

[cerveada]

Hi,
this should be quite easy to do, but there is no tool to do it for you right now.

Both ConsoleLineageDispatcher and LoggingLineageDispatcher are printing the same json that is sent normally via http dispatcher. So you can just take this json manually and send it via http to spline server. You need to provide the same headers that the dispatcher normally sends, but that should be it.

[cerveada]

No, Spline-UI needs Spline Server to query for the data.

[cerveada]

If there is a third party tool to do that, I don't know about it.

[maybenotconnor]

Is spline server able to be pointed to a file? Or can I manually move the output into arangodb?

[cerveada]

Is spline server able to be pointed to a file?

No, the server provides rest endpoint and expects the lineage to be sent there.

can I manually move the output into arangodb?

This would be much more complicated the just sending it to a server, that already knows how to store this in arango.

The server, arangodb and ui are all parts of one app. Trying to just use part of it will make everything more complex and fragile.

---

If you are ok with manual solution, you can just take a JSON produced by ConsoleLineageDispatcher or LoggingLineageDispatcher and send it to the server via http. (HDFSLineageDispatcher does not use json format supported by the server)

[maybenotconnor]

What format is the hdfs dispatcher using? Is there something that can read it? Or if I could make a custom dispatcher?

(thank you so much for your help. I’m just a dev who was told to add data lineage without using an API)

[wajda]

It's a JSON in a Spline format. There is no UI that can open it other than a general purpose JSON viewvers/editors. But hat's probablu not something you are looking for...
What is your problem exactly? Why can't you use REST/HTTP to sent the data to a running Spline server?

[maybenotconnor]

I’m in a cloud environment in which I’m limited to what can be provisioned. Thank you for your help, I have determined I will write a script similar to the producer in
https://github.com/aws-samples/data-lineage-for-data-lake-example
in order to save the data in Neptune, and will just have to visualize using queries

[wajda]

I see. Yes, that's too bad. You will have to basically rewrite a good half of Spline server logic to store, query and visualise lineage graph, especially if you need it end-to-end.
Not sure if it helps you, but Spline can perfectly run on Kubernetes / EKS, just in case it's allowed on your cloud.