Wolfram Engineering: Avoid transferring/converting huge data when not explicitly requested

[holyjak]

Some Wolfram functions such as Import of a 400k CSV or VideFrameList may return huge data, and transferring that and parsing into JLink Expr and than Clojure is slow typically not really desired. In most cases you want to store them into a Wolfram-side variable and not transfer them back.

It would be good to notice that we are trying to transfer huge data and abort / shorten unless explicitly instructed not to. Similarly as println does when *print-length* / depth is set.

The big question is how do we detect this case... Not sure whether it is possible on the JVM side, without going through all the data. Is there perhaps some Wolfram fn that can tell us the "size" of the result, which we could call automatically (or for a list of "dangerous" functions) to find out the size before we start transferring all the data? Or is there some other way?

[light-matters]

An example of generating a lot of data is

  (-> path--video
      w/Video
      (w/VideoTrim [12 24])
     (w/VideoFrameList 50)

[holyjak]

Wolfram Engineering: There is no way to do it on Java side, but you could wrap expression with a tiny piece of Wolfram to achieve this. F.ex. st. like

limitSize[expr_]=If[ByteCount[expr] > 10000,"BEWARE-TOO-BIG!",expr]

Then wrap each expr. with (limitSize <expr>).

It is not cost-free, but likely negligible.