feat: Save/export a running experiment

[sdelliot]

Save/export a running experiment

Description

This draft PR is the start of a conversation around enabling users to save off (or export) a FIREWHEEL experiment. This export can then be loaded back onto the same server or potentially a different server. This solves two primary issues:

1. Rebooting servers during a FIREWHEEL experiment causes issues, but leaving a server running over the weekend is often not necessary and expensive.
2. Enables sharing a running an experiment with others.

There are a few open questions that should be discussed:

Saving VM Schedules

The PR as written, will save all VMs experiment schedule and then loads that schedule back. Generally, this sounds great. However, there are some nuances with FIREWHEEL which make this difficult. First, the entire schedule is collected, not just the portion of the schedule that has not yet run. That means that upon load, the entire schedule will proceed to run again. I have a few thoughts on how we can deal with this.

Option 1: We can prohibit saving the schedule entirely or load in an empty schedule. To me, that would make this feature noticeably less useful and functionally only a wrapper around minimega's ns save.

Option 2: We can "intelligently" save off the schedule. That is, we can save the experiment time, and remove all scheduled events prior to the "current time". In this case, because negative time is per-VM, no negative time events would be saved. Rather, only positive time events that "have not yet run" would be saved. I think that this intelligent saving could happen at the "save point" or firewheel could update the VM's schedule after each time step. The second option may preserve the ability to have negative time resources, but would add a lot of overhead to the system with the various DB updates.

Option 3: We can potentially update the VM resource handler to identify duplicate entries or entries that have happened before. To me, this seems very risky and could break existing expectations of the system.

Issues with minimega

This feature relies on the current master branch of minimega, which has not been formed into a release yet (soon to be v3.0.0). Both the minimega binary and associated Python package will need to be updated prior to merging this PR. However, there are some lingering issues including the discussion at sandia-minimega/minimega#1545 over backing images starting with relative paths.

Experiment time

Should we save the prior experiment time. More importantly, if we do, should we adjust the start time? That is, if we save an experiment with a start time of 0800.00 after 10 seconds (current clock is 0800.10) and we resume the same day at 0900.00. Should the experiment time reflect 0800.10 or 0900.10?

Unfinished Features

• Converting the experiment into a tarball is nice, but should be optional. It can take a very long time for large experiments. Alternatively, we can identify a faster way of compressing the data.
• Saving off experiment start time
• Needs additional documentation and likely a tutorial
• Dependent on minimega v3.0.0

Type of Change

Please select the type of change your pull request introduces:

• Bugfix
• New feature
• Documentation update
• Other (please describe):

Checklist

• This PR conforms to the process detailed in the Contributing Guide.
• I have included no proprietary/sensitive information in my code.
• I have performed a self-review of my code.
• I have commented my code, particularly in hard-to-understand areas.
• I have made corresponding changes to the documentation.
• My changes generate no new warnings.
• I have tested my code.

Additional Notes

Please provide any additional information or context for your pull request here.