Fix blank mse_summary

[imreddyTeja]

The mse_summary.jl script always prints no summaries or error. This is because the reproducibility files are now placed in a separate subfolder, which also contains a prog_state file. This commit updates the get_computed_mses default value for subfolder to reflect this, and ignores the prog_state file when searching for the most current comparison.

closes #3892

---

• I have read and checked the items on the review checklist.

[haakon-e]

I'm probably not the best to review this, so I'll just leave some thoughts:

• Based on your description, the changes seem reasonable.
• Based on your description of how the folder structure of reproducibility files has changed, would it make sense to add a test that verifies the current expectation of things, to ensure that this code is remembered if the folder structure ever changes again in the future?
  • For example, what would happen if we changed the name of the prog_state.hdf5 file?
  • What would happen if we changed the name or contents of the reproducibility_bundle folder?
  • etc...

[trontrytel]

I agree with @haakon-e that it would be nice to add a test that checks if the file structure is as expected. Maybe even add a comment somewhere in the docstring, docs or wiki or wherever people dealing with CI issues will look first

[nefrathenrici]

Thanks for catching this, it looks good to me. I will just echo the other comments that this should fail if no relevant files are found to prevent silent failures in the future.

[imreddyTeja]

I'm probably not the best to review this, so I'll just leave some thoughts:

• Based on your description, the changes seem reasonable.

• Based on your description of how the folder structure of reproducibility files has changed, would it make sense to add a test that verifies the current expectation of things, to ensure that this code is remembered if the folder structure ever changes again in the future?

  • For example, what would happen if we changed the name of the prog_state.hdf5 file?
  • What would happen if we changed the name or contents of the reproducibility_bundle folder?
  • etc...

Thanks for the reviews everyone.

A test is definitely a good idea, and I will add one before merging.
The reproducibility code as a whole contains quite a few hard-coded paths.
This should be solved with Issue #3887