Don't rely on there being an `additional_stores` field (e.g. with Atomate2)

[Andrew-S-Rosen]

The additional_stores field can be used to store large documents, but not everyone needs to do this in practice. If the user's jfremote.yaml does not have additional_stores, Jobflow-Remote crashes with the following:

Traceback (most recent call last):
  File "/scratch/gpfs/ROSENGROUP/asrosen/jobflow/vasp/atomate2/job.py", line 18, in <module>
    ids = submit_flow(flow, worker="basic_vasp")
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/asrosen/.conda/envs/cms/lib/python3.12/site-packages/jobflow_remote/jobs/submit.py", line 80, in submit_flow
    raise ConfigError(
jobflow_remote.config.base.ConfigError: Additional stores ['data'] are not configured for this project.

The proposal here is to accept the user not providing an additional_stores or to default to using the main jobstore for this.

Migrated from materialsproject/atomate2#1260.

[gpetretto]

It could be that in our (or at least mine) understanding the additional_store specified should have mandatory, but it was quite some time ago, so maybe we had it right but maybe we just thought that it would be better to enforce this anyway.

It is easy to remove the check but we also expect that most users would prefer to know that it is advisable to use the additional_store. What do you think about replacing the error with a warning? Or do you think it would still be too annoying?

[Andrew-S-Rosen]

While I think most users, in principle, would want to set an additional_store for storing large data files, in practice this is often very difficult in an academic context. It is often difficult enough to have access to MongoDB for the JOB_STORE, but having some larger data service for storing blobs is often not trivial. At least, that's how I am assuming the additional_store is meant to be used (please correct me if I'm wrong). In that scenario, we end up just setting it to a MemoryStore.

Maybe one option would be to have a warning (but not an error) when doing jf project check? A warning every time a job is run seems like it might be a bit much. I'm open to your suggestions.

[davidwaroquiers]

I would guess that if you anyway have an access to MongoDB, you can set the additional store as a gridfs store on the same mongodb instance no ? (I think even setting another MongoStore, e.g. with a different collection, would also work). Nevertheless, we can remove the raise of the error there and add it in the jf project check.

[JaGeo]

We always set it up as an Gridfs store! This works easily.

[davidwaroquiers]

Yes Gridfs is very easy (also, is much better than standard mongostore for the additional store as it does not have the 16Mb limit in case of large objects).
Would that work out for you @Andrew-S-Rosen ? (it does not prevent us from moving away the check+error that you observed in submit_flow to a warning in jf project check)

[gpetretto]

While I think most users, in principle, would want to set an additional_store for storing large data files, in practice this is often very difficult in an academic context. It is often difficult enough to have access to MongoDB for the JOB_STORE, but having some larger data service for storing blobs is often not trivial. At least, that's how I am assuming the additional_store is meant to be used (please correct me if I'm wrong). In that scenario, we end up just setting it to a MemoryStore.

I was also thinking that once a MongoDB is available setting up a gridfs store would be quite straightforward. But I understand that storing large amount of data could still be problematic. Maybe it would interesting to devise some way to tune the amount of data saved directly in jobflow/atomate2?

Maybe one option would be to have a warning (but not an error) when doing jf project check? A warning every time a job is
run seems like it might be a bit much. I'm open to your suggestions.

The problem with checking in jf project check is that the name of the additional store is defined on the Job. So in the check we can suggest that an additional_store is not defined, but 1) it may not be needed 2) the name is not known, so we could say that the one for atomate2 is missing, but it is not necessarily the correct one.

I agree that a warning for every submitted Flow may be annoying. We could decide to add an option in the Settings to disable the warning, but one would still need to know that this option exists...

[davidwaroquiers]

The problem with checking in jf project check is that the name of the additional store is defined on the Job. So in the check we can suggest that an additional_store is not defined, but 1) it may not be needed 2) the name is not known, so we could say that the one for atomate2 is missing, but it is not necessarily the correct one.

True ...

Maybe the option in the Settings would be good. About knowing whether the option exists, this could be provided in the warning itself. Something like:

Warning: jobflow_remote.config.base.WARNING: Additional stores ['data'] are not configured for this project. If you want to disable this warning, you can set this XXX in the SETTINGS.

(Note that this setting could be: None, 'warning' or 'error' and by default we could set it to 'warning').

[Andrew-S-Rosen]

Thank you all for the very useful input and @JaGeo for sharing your experience!

I naively had not used the GridFS Store because I figured we would run out of space, but perhaps that is not a major concern for us in practice. I guess this points to a need for better documentation in Atomate2 or Jobflow about what these stores are and how they are used --- I wasn't actually aware that GridFS was associated with MongoDB!

As for this issue report, I think we should keep things as simple as possible. I think a global setting might be a bit cumbersome. We should probably try to keep those as few as possible for the sake of the user. Perhaps it is fine as-is and we would be better off with more documentation in Atomate2.