Switch Domain.java constructors to use ephemeral TDB databases instead of in-memory

[ibaldin]

An Exo controller slowdown triggered investigation into memory use. Heap dump showed that active slices were consuming large amounts of RAM (on the order of 100M each) by storing in-memory ad models acquired from the broker. The problem appears in Domain.java constructors that create a model in-memory.

An initial trivial attempt to switch those to TDB Ephemeral resulted in

• Jenkins running out of memory on tests (!)
  and
• Tests by @hinchliff showing a significant degradation in performance (a slowdown factor of 3-4 in tests).

We should investigate this further, this thread is the place to dump findings and conclusions

A related issue in controller JVM configuration was discovered that limited actual heap to 4G (see RENCI-NRIG/exogeni#143) using Tanuki wrapper.

@YufengXin

[hinchliff]

Here is a link to the the test failure in Jenkins: persistent-domain-model/1/

I tried replicating this by running the maven tests on my local box, using the git commit with TDB Ephemeral: ac7d9d

I wanted to see if increasing the maximum heap size for the maven surefire test plugin would eliminate the Heap and GC errors that Jenkins produced.

Running ac7d9d locally without any Heap size modifications strangely resulted in the tests effectively never finishing. I think I saw some test failures, but without explicit messages as to what caused them. My laptop became very unresponsive, and after over an hour I killed the test. I haven't been able definitively confirm, but I think the tests were limited to 4GB of Heap.

I changed the maximum heap size to 8GB, and ran the tests again. The tests were still slower than normal, but ended in failure after 25 minutes with Heap and GC errors.

[hinchliff]

To check next week:

can you try setting uniqueDM to true (create a document manager for each model separately)? It's parameter on a call to getModelFromXXX currently set to false.

[ibaldin]

Tried UniqueDM and other fixes - results are same - the tests never finish.