Create differential tests on output data prior to merging MPI and OpenMP solutions

[dorchard]

(This issue is to start a discussion on testing strategies)

Once we get to the point of merging MPI and OpenMP parts of the code base it would seem useful to have end-to-end tests that can compare output runs to some 'golden outputs' so that we can catch any problems, and assess with any changes are within acceptable error bounds (e.g., due to re-association of float point operations coming from parallelisation). Calculating RMSE (or something similar) between the output and the golden output would seem a useful measure here (we can also just do bitwise comparison as well - although I suspect that meeting that would be optimistic). This would generally help I think with code optimisations. Sometimes the golden outputs would need to change (due to scientific reasons) but other times its good to have them to assess changes that we want to be semantics preserving (e.g., optimisations).

[dorchard]

This issue depends on #334.