Best practices in file management

[Andrew-S-Rosen]

This Q&A stems from a conversation with @blaked8619, who I will let expand upon my query here as he sees fit.

In short, the question is: are there any recommended "best practices" for dealing with file management and Jobflow-Remote? With FireWorks, users had to ignore their desire to organize files and instead rely on everything being stored in a single directory for a given project. The philosophy was that the user should solely rely on querying their database and that smart queries would take the place of an organized filesystem. Of course, in practice, this all-or-nothing approach proves quite difficult for many users.

Is Jobflow-Remote intended to be used in the same way as FireWorks in this regard? One can change the work_dir for the worker and make a unique worker for each project. This is fine, but if a user wants to organize files within a given project, it is less clear if there is a way to do that. For instance, perhaps a user wants to store all PBE jobs in a folder named PBE and all r2SCAN jobs in a folder named r2SCAN. Or perhaps a user wants to store all phonon jobs in a folder named phonons and all relaxations in a folder named relaxes.

Naturally, the user should not be moving files after the jobs are run because this will cause a mismatch between the job's metadata and where it is actually stored.

[JaGeo]

I haven't found a food solution so far as well:
There are several points that complicate the organization/file management for me.

• it's hard to remember the script used for the submission of the flow as I submit quite complex experiments sometimes. at the moment, i use the script names as job names. Yes, i could add further meta data but then again it is complicated to identify the flow easily.
• if you need to update the code, you need to restart and it could disrupt old workflows (new project in such a case). I would make a project for a set of calculations that you want to run with one code version
• if you use a remote cluster, often they are not backed up. Thus you regularly need to collect and move all data. While it would be ideal to keep it in place, it is not always practical. I would still be really happy about an option to retrieve all data for one flow.
  No real answers. Rather more questions 😅

[Andrew-S-Rosen]

All good points. Thanks for chiming in!!

if you use a remote cluster, often they are not backed up. Thus you regularly need to collect and move all data. While it would be ideal to keep it in place, it is not always practical. I would still be really happy about an option to retrieve all data for one flow.
No real answers. Rather more questions 😅

We are in a similar situation here. We run in a high-performance I/O directory, but that directory is not backed up. Moving the data after job completion will make the job metadata inaccurate (and also any directory metadata in the output task document inaccurate). Since our high-performance I/O directory (the runtime directory) is not purged, we currently have an rsync-like service set up to sync between the runtime directory and a more permanent file storage location. We still do most of our day-to-day activities out of the high-performance I/O directory to not break the metadata but with the expectation that it might be lost one day and we will need to recover data from our synced mirror, which is backed up. It's not an ideal setup, but it's what we have going on right now.

[JaGeo]

In the next year, I would like to come up with a long-term solution for data management in the group. Thus, I am happy to hear what other people are doing or planning to.

[JaGeo]

Ideally we had a flag that enables automatic download of the files at the end of a flow and an update of all directories in the meta data.

[Andrew-S-Rosen]

Ideally we had a flag that enables automatic download of the files at the end of a flow and an update of all directories in the meta data.

This seems fairly reasonable --- perhaps something that could be specified in the exec_config of the submit_flow() call where the user could specify something akin to a final_dir.

[Andrew-S-Rosen]

And in case it helps your group at all, we use a program called Syncovery to automate our file transfers/syncs. I've used it for ages since it's cheap (<$100) and seamless. Plus they are German. 😉

[gpetretto]

Thanks both for bringing up this topic. From what I see there could be two different kind of issues to be discussed:

1. More control on the directory of execution
2. handling of the backup

so I will try to approach these separately, but feel free to point out if I may have missed some connection among the two in your use cases

Control work_dir

As pointed out by @Andrew-S-Rosen, the idea of jobflow-remote was to somewhat follow in firework's footstep regarding this point, thus focusing on the interaction with the DB.
However, the work_dir is set in the DB by the Runner during the "upload" phase and then the value from the DB is used for subsequent actions. In this view it would be quite easy to define different execution directories based on Flow and Job names. For example a choice could be to have a folder like this:
<worker's work_dir>/<Flow name>/<Flow uuid>/<Job name>_<Job uuid>_<job_index>
This means that if you name your flows "relax" or "phonons" all the similar calculations will be collected in the same subfolder.
Another alternative could be a work_dir attribute for the Flow in the queue store and that folder will be used as base folder:
<Flow work_dir>/<Job name>_<Job uuid>_<job_index>
And then it would be up to the user to select the base folder for each flow (it can be an argument for submit_flow).

Base folders could also be defined based on the Maker's name of the Job, if any. This will allow to group the same kind of jobs together.
Multiple options can implemented and this should be relatively fast to do, but I would define a general switch in the project configuration rather than defining this on a per Job basis.
I think this could already cover a certain number use cases.
Instead, creating something that tries to collect jobs based on some logic extracted from the content of the Job would be more complicated and I am not sure how this could be done exactly if jobflow-remote remains agnostic with respect to the type of jobs that could be executed.

Backup

How to backup data after a Flow has finished is also a topic we have been discussing for some time now.
My view is that there are many different use cases and expectations of what a backup tool can or should do. There are also many tools and locations where the data could be moved to. It turned out that people may need to move data to specific folders, remote servers, services hosted by the cluster or even cloud services (AWS, Azure). At some point even Nomad was mentioned as final destination for the raw data. Depending on the use cases, these targets may be accessible only from the cluster or only from the machine hosting the Runner. As a consequence I have started doubting that an automated one-size-fits-all solution could be actually designed.
So, I was thinking of rather focusing on tools/functionalities that would help creating backups rather than providing a fully automated solution that will necessarily have a limited number of options. To be more explicit, some examples could be:

• I would like to add an option to provide to a Job the list of all the folders of the other jobs in the Flow. The idea would be that one can then implement a "final" Job that copies all the data for a Flow as a last step.
• Making the JobController available to the Job if it runs on a "local" worker in case some updates need to be done to the job folder (together with full information about all the steps of the flow)
• add an option to define a callback function associated to the completion of the Flow that will be executed by the runner (this might be a bit tricky: where should this be kept track of? what if it fails?)

In general, the main idea would be provide similar tools, but leave the fine details of the implementation of the backup jobs or callback functions to the user, so that anyone can define: the kind of data to backup, where the backup procedure is executed, where and how the files are copied.

In any case, nothing has been decided on these topics, so the discussion is open.

[JaGeo]

Regarding backup:

The final job option raises some questions for me: if it is a job, the job needs to be able to copy from the remote server. At least with two-factor authentification, this would be tricky.

[Andrew-S-Rosen]

Regarding file organization: I am in strong support of your suggestions, both in terms of organizing based on flow and job names as a sensible option and also custom details with the submit_flow call. These would be excellent additions!

[gpetretto]

Regarding backup:

The final job option raises some questions for me: if it is a job, the job needs to be able to copy from the remote server. At least with two-factor authentification, this would be tricky.

I suppose you are imagining running this Job on the same machine where the Runner is being executed. Would it be an option to run it directly on the worker instead? Or it is not possible to reach your backup location from the cluster?
If that's the case, potential solutions can include:

• Some time ago I started considering a "daemon worker": Deamon worker #75. Meaning a worker where Jobs are executed directly by the Runner and not by external processes. I was thinking this more as a way to execute small and quick jobs, but one can imagine giving this Jobs access to the connections to the other workers and use this to perform backup actions.
• As mentioned above, a "callback" function (or Job?) that can be defined by the user and executed by the runner upon workflow completion.

Do you think that these could be a solution for your use case? There are some technical difficulties for both of them to be taken into account, but I don't see any blocking point in implementing them.

Regarding file organization: I am in strong support of your suggestions, both in terms of organizing based on flow and job names as a sensible option and also custom details with the submit_flow call. These would be excellent additions!

Good to hear that this could be interesting. I will see if/when I have the chance to add them. Or if anyone is interesting in giving a try at implementing this we can discuss more details. In any case more comments on the topic are welcome to check what would be the best way of implementing this.

[JaGeo]

@gpetretto Yes, indeed, the daemon worker would be a very good solution for us. We also do not have access from the remote cluster to our backup location.