bigmemory (or similar method) for large matrix storage

[ericbair-sciome]

Utilizing the correlation based distance can require storage of the full covariance matrix. We will explore ways to ensure it is scalable for n > 60K.

https://fbertran.github.io/bigalgebra/

[kyle-messier]

a def v2.0

[kyle-messier]

@ericbair-sciome Another option to consider for 2.0 is Apache-Arrow. Fast and efficient read/write combined with adapting how the Vecchia neighborhood is stored and updated could help.

[kyle-messier]

We'll want to analyze areas in PrestoGP where memory is built up, opportunities where off-loading memory in exchange for I/O could be used.