Feature Request: Documentation of Slurm JobID

[blaked8619]

Currently, I don't believe that jobflow-remote documents the Slurm JobID associated with a given job or flow DB id. This can make debugging related to failures in job submission on remote machines difficult. Is this something that could be included in .jfremote/log?

[gpetretto]

Hi @blaked8619,

the slurm job id is stored in the DB:

jobflow-remote/src/jobflow_remote/jobs/data.py

Line 125 in 36da668

process_id: Optional[str] = None

You can see it with jf job info under the remote field.

[blaked8619]

@gpetretto I see that now, thank you!

[blaked8619]

@gpetretto , I just noticed that the 'remote' field is different depending on if it is a single submission or a batch job.
For the single submission:
remote = {'step_attempts': 0, 'process_id': '42122129', 'prerun_cleanup': False}

For the batch submission:
remote = {'step_attempts': 0, 'process_id': '4953106d-f831-41c5-a927-8cf762807c50', 'prerun_cleanup': False}

The process_id for the single submission, 42122129, is the slurm JobID I was looking for. However, it seems that this does not show up for the batch submission.

I think this was planned to be addressed in jobflow_remote/jobs/run.py:

TODO the ID should be somehow linked to the queue job
if not batch_manager:
batch_manager = LocalBatchManager(files_dir=files_dir, process_id=process_uuid)

[gpetretto]

I see. Indeed the case of the batch execution is handled in a different way. Despite the TODO, I think that the run is not the suitable point to get the SLURM id. The overall problem is that I need a share unique id that can be know before the SLURM job is submitted, since it needs to be included in the command that will be executed in the submission script itself.
Probably the easiest option would be to set the process_id to something like SLURMID::UUID (e.g. 42122129::4953106d-f831-41c5-a927-8cf762807c50). I will try to see if this can work without altering much the current code. Do you think this would help you?

For the moment I can also point to the jf batch list command. I am not sure if it will be helpful in your case, but this prints the list of the current running batch SLURM jobs. This contains both the identifier uuid set in the process_id and the SLURM id. However, the entry is deleted when the SLURM job is finished and there is no history of these jobs. I considered keeping such an history, but this would have required a dedicated collection and it seemed a bit too much.

[Andrew-S-Rosen]

Probably the easiest option would be to set the process_id to something like SLURMID::UUID. Do you think this would help you?

I think this would be perfect.

For context, one of the reasons we need the Slurm ID is that we occasionally have machine failures or other issues to report to the HPC staff, and it is necessary in such cases to have the Slurm ID for troubleshooting purposes.