Running Multiple Projects to Different Database Collections/ Running Multiple Projects on One Daemon

[blaked8619]

I am looking to run multiple jobflows (projects) that point to different collections for storage in mongodb. Since I want to store different jobflows in different collections, defining a single project with two different workers is not sufficient, since both of these workers would point to the same collection. I think one way to solve this would be defining two different projects (different .yaml files) that each have their own workers and collections defined. However, it seems that one daemon isn't able to cover both projects:

(cms) [bd8619@tiger-arrk .jfremote]$ jf -p cms runner status
The selected project is cms from config file /home/bd8619/.jfremote/cms.yaml
Daemon status: running
(cms) [bd8619@tiger-arrk .jfremote]$ jf -p stellar runner status
The selected project is stellar from config file /home/bd8619/.jfremote/stellar.yaml
Daemon status: shut_down
(cms) [bd8619@tiger-arrk .jfremote]$ jf -p stellar runner start
The selected project is stellar from config file /home/bd8619/.jfremote/stellar.yaml
⠋ Starting the daemon...Error while starting the daemon: A daemon runner process may be running on a different machine.
Here is the information retrieved from the database:
- hostname: tiger-arrk
- project_name: cms
- start_time: 2025-07-23 13:32:03.142000
- last_pinged: 2025-07-23 13:32:03.142000
- daemon_dir: /home/bd8619/.jfremote/cms/daemon
- user: bd8619
If no runner is active on that machine clean the DB with `jf runner reset`

Is there perhaps another more practical way to run two different jobflows to two different collections at once?

[gpetretto]

Hi @blaked8619,

I see what your problem is in the example that you posted, but I am not sure I understand what you would like to achieve ultimately.

In the error message that you report the problem is likely that you created two projects, but you are pointing to the same queue database for both of them and are just using two different jobstore. This is not possible, since in the queue collection there is no information about different projects. In this context, even if you were able to start two different runners, these will just be in a race condition to pick up different jobs and they would end up in different output databases randomly. For this purpose we introduced the check and try to prevent multiple runners starting on the same queue DB.

So, if you just want to have two different projects you also need to define two separate queue DB, one for each project. Then you can start the two runners separately. Currently each runner is limited to pick jobs from a single project, so there is currently no "master runner" that can handle multiple projects simultaneosly. In principle I think it might be feasible to create one, but I am not really convinced that it would be particularly beneficial while there are definitely some downsides to that.

It is not clear if instead you would like to have a single project with different stores and have the output of different Flows be directed to different stores. (Note that for how jobflow itself works at the moment, it will be impossible to use multiple JobStores as outputs for a single Flow, since only one Store can be used to resolve references of previous steps). It should be possible to implement an option to assign different stores to some Flows, but it is not available at the moment.

[davidwaroquiers]

Hi @blaked8619,

I would add that if it's easier (and @gpetretto's answer is not enough), we may have a short call to properly understand what you are trying to do.

[blaked8619]

Thank you @gpetretto and @davidwaroquiers, you addressed everything I was questioning about the problem. As for why I wanted to do this, @gpetretto you are correct in saying I would "like to have a single project with different stores and have the output of different Flows be directed to different stores". I would like to do this for organization purposes, as I have jobs of different types that I would like to store in different collections. I understand that the alternative to this would just be tagging each job with a different name or metadata then storing all jobs in the same store/collection. My personal preference is the collection-separated route as it is a more intuitive folder or directory-like way of storing jobs.

[gpetretto]

Thanks for clarifying the issue.
We discussed earlier with @davidwaroquiers and we think that this could be done in the following way:

• adding a section to the configuration file, allowing to insert other jobstores

  alternative_jobstores:
    another_jobstore:
        docs_store:
          type: MongoStore
          database: test
          host: localhost
        additional_stores:
          data:
            type: GridFSStore
            database: test
            host: localhost

  (probably with a different name than alternative_jobstores)
• adding a store option to submit_flow so that a Flow can be submitted to

  submit_flow(myflow, worker="xxx", store="another_store")

Then all the Flows that have been submitted in this way will use the selected alternative Store. Do you think this will suit your use case?
There might be some techinical issues that could pose some problems, so in case I cannot guarantee if and when we could implement this.

[Andrew-S-Rosen]

Just chiming in to say that I think such a feature, namely being able to specify different stores at submit_flow time, would be excellent!

[blaked8619]

I agree, this feature would be great!