Error handling of slurm jobs

[staeglis]

At the moment run.sh just checks the output of the error log: If this isn't empty the CI task is handled as failed. This causes that all CI pipelines using slurm are failing although the status of the job is "COMPLETED" and not "FAILED".

Can run.sh check the status code of the job instead?

[tcojean]

This question of failure is a bit complicated. The current approach clearly needs to be improved and is not satisfactory for very general usage. The current approach is also about clearly identifying error output, but some applications use stderr for information rather than actual errors. Maybe #3 is another way to fix this? The job status way could be another approach if we can extract the information effectively.

[staeglis]

The job status is checked already via slurm_wait_for_status, isn't it?

[tcojean]

The status there is only the SLURM status, whether it is COMPLETED, TIMEOUT, CANCELLED, FAILED, and more. AFAIK, this does not necessarily equate to error status, hence the stderr based system.

[staeglis]

It depends. I would suggest to check the exit code for each command of the ci scripts like this is done in the docker runner.

You could insert lines like this:

if [ $? -ne 0 ]; then exit 1; fi