Simplifying the installation process?

[JaGeo]

I think a huge problem with using tools like jobflow-remote is still the installation process.

Do you have any ideas how we could simplify this?

Is there a way to provide a conda environment/a docker including database installation (maybe even with a default setup of the database)?

We just got another round of questions on the atomate2 repo.

[Andrew-S-Rosen]

A docker container is an interesting idea. I think the biggest challenge is the database. If someone were able to code up a file-based Maggma store (even if it is not performant!) that supports parallel processes as noted in materialsproject/maggma#832, it would be much easier to describe the installation process. But as it stands, by essentially relying on MongoDB, the barrier for entry will remain high unless it can be solved with a container.

I will note that this is something Prefect does very well. They ship everything you need in a Conda environment or as a container, and it "just works." The default database configuration is not very performant, but for testing purposes, it works just fine.

[davidwaroquiers]

I think it would indeed be a good idea to have something like that. It would definitely lower the barrier to entry when someone wants to try it out first. There is one thing I would do though is to explicitly add a warning all the time when you are using such less-performant store (or implementation - currently jobflow-remote requires a real mongodb database because it uses aggregations, which are not supported by current file-based Stores - I don't think it would ever be btw). Otherwise some users may think "oh great, I don't need a database in production". I think anyone serious about doing high-throughput production should, after having tested with the less-performant store, should take the time to set his production environment up with a real database.

Warning could be:
"You are using a mock database with limited performance and reliability. This database is provided only for testing and training purposes. For production, you must set up a real MongoDB database as explained ."

I'm not necessarily in favor of a container. It can be quite some maintenance headache, the "it just works" promise is not always (duh) satisfied.

I think it would be possible to have a fallback implementation that does not use aggregations (@gpetretto ?) but with 1/ limited performand (of course) and 2/ (more important) limited reliability because of possible concurrent accesses. (hence the warning that should be there all the time)

What do you think ?

[JaGeo]

I think a typical use-case that we do not cover yet:
I want to run one elastic or one phonon workflow for a large structure but I don't want to setup/configure a database myself for this.

[gpetretto]

Hi,
thanks for raising this issue.
I actually think that there are several points where a user may get lost in setting things up. Unfortunately, I don't think it will be easy to provide something ready to use for all of them, but at least we can try to make it easier.

For the database, indeed we could prepare a further simplified version of the docker container in https://github.com/Matgenix/jobflow_tutorial, removing the local worker and just leaving the runner machine plus MongoDB. However, this will work only if they can run jobflow-remote on a local workstation. If they want to have the runner on the front-end of the cluster they will either need to be able to reach the DB from the cluster (which likely does not include a personal workstation where docker can be executed) or running the DB directly on the cluster frontend (which I would not really recommend for a beginner. There is definitely a possibility that cluster administrators will complain or to expose the data to other users, if the DB is left without any authentication).
We could prepare a predefined configuration for the free tier of MognoDB Atlas, but this could only work for a short amount of time, as performances are not very good and the disk space will be over quite soon.
As already mentioned, an alternative solution would be to devise a way to use file-based DB, but this is not so trivial. I keep thinking about it and the problem is mainly to identify the right tool to use, that will not be abandoned in a short time. I am still making some tests on this.

However, once the DB is set, one still needs to perform some minimal setup of the project configuration to point to the worker. A thing that is strictly dependent on the user HPC system. @Andrew-S-Rosen, I suppose that this is true for Prefect as well. Or do you really have the feeling that you can easily run on HPC systems out of the box? In that case, how is that achieved?
For jobflow-remote we can maybe provide some tool to automatically fill in a project configuration file? But will still require some work from the user.

Finally, a user needs to understand how to actually use the tool: submitting a Flow properly with the correct resources, monitor the status of the Jobs, extracting results and maybe even check the errors when something goes wrong. My experience with newcomers to workflow systems is that learning how to perform these actions is still something that generate a lot of doubts, even after providing individual training.
I don't think this is strictly related to jobflow-remote, as I had the same kind of experience with new Fireworks users and even on myself when I have used Aiida. Despite all the experience, the differences in setup/approach/commands required me to consult the documentation quite often.

My main point here is that we should definitely try to make things smoother for the generic user, even if I doubt we will ever manage to make something as simple as the run_locally within jobflow-remote or fireworks. But it should be somewhat clear that the user should be available to put in some efforts in learning the basics of the tools. I have the feeling that scientific workflows are often sold as an effortless way of running the simulations and thus some users may end up having the wrong expectations of the amount of work they will still need to put in learning how to properly use them.

[Andrew-S-Rosen]

@Andrew-S-Rosen, I suppose that this is true for Prefect as well. Or do you really have the feeling that you can easily run on HPC systems out of the box? In that case, how is that achieved?

You are correct. The interfacing with the job scheduler remains a learning curve for every single one of the 300+ workflow systems published to date, including Prefect (it uses dask-jobqueue to interact with the scheduler, which is terrible).

[JaGeo]

Hi, thanks for raising this issue. I actually think that there are several points where a user may get lost in setting things up. Unfortunately, I don't think it will be easy to provide something ready to use for all of them, but at least we can try to make it easier.

For the database, indeed we could prepare a further simplified version of the docker container in https://github.com/Matgenix/jobflow_tutorial, removing the local worker and just leaving the runner machine plus MongoDB. However, this will work only if they can run jobflow-remote on a local workstation. If they want to have the runner on the front-end of the cluster they will either need to be able to reach the DB from the cluster (which likely does not include a personal workstation where docker can be executed) or running the DB directly on the cluster frontend (which I would not really recommend for a beginner. There is definitely a possibility that cluster administrators will complain or to expose the data to other users, if the DB is left without any authentication). We could prepare a predefined configuration for the free tier of MognoDB Atlas, but this could only work for a short amount of time, as performances are not very good and the disk space will be over quite soon. As already mentioned, an alternative solution would be to devise a way to use file-based DB, but this is not so trivial. I keep thinking about it and the problem is mainly to identify the right tool to use, that will not be abandoned in a short time. I am still making some tests on this.

However, once the DB is set, one still needs to perform some minimal setup of the project configuration to point to the worker. A thing that is strictly dependent on the user HPC system. @Andrew-S-Rosen, I suppose that this is true for Prefect as well. Or do you really have the feeling that you can easily run on HPC systems out of the box? In that case, how is that achieved? For jobflow-remote we can maybe provide some tool to automatically fill in a project configuration file? But will still require some work from the user.

Finally, a user needs to understand how to actually use the tool: submitting a Flow properly with the correct resources, monitor the status of the Jobs, extracting results and maybe even check the errors when something goes wrong. My experience with newcomers to workflow systems is that learning how to perform these actions is still something that generate a lot of doubts, even after providing individual training. I don't think this is strictly related to jobflow-remote, as I had the same kind of experience with new Fireworks users and even on myself when I have used Aiida. Despite all the experience, the differences in setup/approach/commands required me to consult the documentation quite often.

My main point here is that we should definitely try to make things smoother for the generic user, even if I doubt we will ever manage to make something as simple as the run_locally within jobflow-remote or fireworks. But it should be somewhat clear that the user should be available to put in some efforts in learning the basics of the tools. I have the feeling that scientific workflows are often sold as an effortless way of running the simulations and thus some users may end up having the wrong expectations of the amount of work they will still need to put in learning how to properly use them.

I think I would cover these two scenarios with a simplified Installation (ideally with a installation script)

• local workstation with MongoDB (preferred locally) that can access a remote cluster

• Installation on the cluster frontend

A local linux workstation is likely the easiest thing to configure for people and for non-high-throughput works it should be fine.

I agree with the other points but tutorials will help there. After all, we are talking about people that figured out how to run/compile DFT codes on HPCs