`OnMissing.NONE` not being passed on to children of dynamic flows

[osoulas]

Hi there,

I have been working on some flows that catch jobs that time out and noticed that some of my child jobs were stuck as 'Waiting' even though I'd set completion_checker_job.config.on_missing_references = OnMissing.NONE. @utf and I have worked out that this is because the config isn't being passed on to the children of completion_checker_job. I have written a minimum working example below:

from jobflow import Flow, Response, job, OnMissing
import time

@job
def double(a):
    if a == 1:
        time.sleep(10000)  # Simulate some jobs taking too long to complete
    else: 
        return a * 2
    
@job
def controller(a):
    job = double(a)
    completion_checker_job = completion_checker(job.output, a)
    completion_checker_job.config.on_missing_references = OnMissing.NONE
    flow = Flow(jobs=[job, completion_checker_job], output=completion_checker_job.output)
    return Response(replace=flow)


@job
def completion_checker(output, a):
    if output is None:
        controller_job = controller(2)
        return Response(replace=controller_job)
    else:
        return output


controller_job = controller(a)

This can be fixed by setting the config for all child jobs:

from jobflow import Flow, Response, job, OnMissing
import time

@job
def double(a):
    if a == 1:
        time.sleep(10000)  # Simulate some jobs taking too long to complete
    else: 
        return a * 2
    
@job
def controller(a):
    job = double(a)
    job.config.on_missing_references = OnMissing.NONE
    completion_checker_job = completion_checker(job.output, a)
    completion_checker_job.config.on_missing_references = OnMissing.NONE
    flow = Flow(jobs=[job, completion_checker_job], output=completion_checker_job.output)
    return Response(replace=flow)


@job
def completion_checker(output, a):
    if output is None:
        controller_job = controller(2)
        controller_job.config.on_missing_references = OnMissing.NONE
        return Response(replace=controller_job)
    else:
        return output


controller_job = controller(a)

[gpetretto]

Hi @osoulas,
thanks for reporting this.
The source of the behaviour you are observing is clear from the Flow diagram of the example you shared:

flowchart TD
    classDef WAITING fill:#aaaaaa
    classDef READY fill:#DAF7A6
    classDef CHECKED_OUT fill:#5E6BFF
    classDef UPLOADED fill:#5E6BFF
    classDef SUBMITTED fill:#5E6BFF
    classDef RUNNING fill:#5E6BFF
    classDef TERMINATED fill:#5E6BFF
    classDef DOWNLOADED fill:#5E6BFF
    classDef REMOTE_ERROR fill:#fC3737
    classDef COMPLETED fill:#47bf00
    classDef FAILED fill:#fC3737
    classDef PAUSED fill:#EAE200
    classDef STOPPED fill:#fC3737
    classDef USER_STOPPED fill:#fC3737
    classDef BATCH_SUBMITTED fill:#5E6BFF
    classDef BATCH_RUNNING fill:#5E6BFF
    2(double) --> 3(completion_checker)
    2(double) --> 5(controller)
    3(completion_checker) --> 4(store_inputs)
    5(controller) --> 4(store_inputs)
    3(completion_checker) -.-> 4debf5b2-ea9f-411d-8ec1-71911a52aa8f
    1(controller) -.-> cc7b1737-cfdd-4ea5-af0c-245329dd3238
    1:::COMPLETED
    subgraph cc7b1737-cfdd-4ea5-af0c-245329dd3238[ ]
        2:::FAILED
        3:::COMPLETED
        4:::WAITING
        subgraph 4debf5b2-ea9f-411d-8ec1-71911a52aa8f[ ]
            5:::WAITING
        end
    end
    style cc7b1737-cfdd-4ea5-af0c-245329dd3238 fill:#2B65EC,opacity:0.2
    style 4debf5b2-ea9f-411d-8ec1-71911a52aa8f fill:#2B65EC,opacity:0.2

Loading

The reason lies in the way the connections between the jobs are defined for dynamic jobs. The idea is: when a Job is replaced the replacing Job/Flow inherits the same parents as the Job that is being replaced. So here double is still a parent of the second controller. In general I think this seems a reasonable choice, but I also understand that this use case is somewhat unexpected. Maybe we can consider changing the logic of the connections for the dynamic jobs.

[davidwaroquiers]

It's also more related to jobflow itself no ?

[gpetretto]

No, if you use run_locally it runs completely. Jobflow does not have the strict concept of parents/children. The only link is if there is a reference to the another job, which in this case is not there. In jobflow-remote it goes a bit beyond that, because there are also other actions to handle (e.g. rerunning a Job with parents/children). Maybe this additional link can be removed (unless the children depends explicitly on the previous job). I would need to investigate further the potential consequences.

[utf]

Hi @gpetretto, thanks so much for investigating this.

To provide a bit more context - @osoulas is trying to create a mechanism to allow incomplete DFT relaxations to finish over multiple jobs. I know we've spoken about actual mechanisms to handle this within JF/JFR but his approach is a more pragmatic until we have a full solution.

Personally, I would have expected his code to run since there is no explicit dependence on the parent job that failed. The work around suggested isn't too difficult to implement but maybe this will be confusing for other people. I don't think this is an immediate priority given the unintended consequences but nice to see the conversation here.