The Philosophy of Datadance

[sripravan]

I wanted to record as part of this discussion that Datadance started as an idea by @karthikchiru12, where he wanted a safe and easy way to manipulate and transform data.

As part of the same vision, I know @karthikchiru12, that we have been intentional about some decisions like:

1. Embedding errors in the output instead of throwing an error.
2. The input object itself should not be mutated but a copy of it which is the derived object can be mutated.
3. We should not allow people to write any Javascript code that they want but instead use an expression evaluator like Jexl.
4. Preserve the order of expression evaluation by using an array of objects instead of an object directly.

Before we launch a v1 of Datadance, I want us to be more intentional about the philosophy of Datadance and document it. This will help us decide the future of Datadance and also help any future contributors to follow the same philosophy.

[sripravan]

I think one of the aspects that I wanted to discuss in this thread is that custom transforms should also be as pure as possible and not do multiple things out of the box. @karthikchiru12 I know you have a different opinion on this, so please add that here, so that we can have it documented for future reference.

[karthikchiru12]

@sripravan I gave it a lot of thought, a magic function really saves a lot of time and takes out the verbosity from the expressions.

But a pure function is more deterministic as it is supposed to do only one thing. Until now the standard library we have built, most of them are not pure functions.

From a design point of view, having pure functions is what sounds right, So from now on, We should make sure that most of the standard library functions are pure.
But there are a few exceptions like eg : forEach, which cannot be converted to pure, so they will stay as is.

I would create an issue, and will start working on converting the existing non pure functions to pure functions, before the v1.0 release.

[karthikchiru12]

One more thing to discuss is that, the documentation is not part of the main repo of Datadance, I would suggest to keep it like that, but @sripravan you were suggesting a mono repo, (excluding documentation, today Datadance follows mono repo style) including documentation. So can you detail how it would help us.