slivka should report the reason that a job was rejected after it was accepted

[foreveremain]

Slivka rejects jobs for a few different reasons. Most of these are at submission time - with a 422 status code in response to the POST. When there are no slivka runners available to accept the job, however, a job can later on be marked rejected.

Right now there's no way to report the reason for rejection - it probably ought to be written into the db and reported in the response, e.g.:

curl https://www.compbio.dundee.ac.uk/slivka/api/jobs/Z8B_rXSW7n_yviqt                   

responds currently with:

{"@url":"/slivka/api/jobs/Z8B_rXSW7n_yviqt","completionTime":null,"finished":true,"id":"Z8B_rXSW7n_yviqt","parameters":{"dealign":null,"full-distance":null,"full-distance-iteration":null,"input":"Z8B_rXSW7n_yviqs","iterations":"1","max-guidetree-iterations":"1","max-hmm-iterations":"1"},"service":"clustalo","status":"REJECTED","submissionTime":"2025-02-27T15:07:25"}

but could instead be:

{"@url":"/slivka/api/jobs/Z8B_rXSW7n_yviqt","completionTime":null,"finished":true,"id":"Z8B_rXSW7n_yviqt","parameters":{"dealign":null,"full-distance":null,"full-distance-iteration":null,"input":"Z8B_rXSW7n_yviqs","iterations":"1","max-guidetree-iterations":"1","max-hmm-iterations":"1"},"service":"clustalo","status":"REJECTED",statusDetail:{"Too many sequences.","No runner could accept this job."},"submissionTime":"2025-02-27T15:07:25"}

[foreveremain]

Looks like a patch is needed just here:

slivka/slivka/scheduler/scheduler.py

Line 170 in 48002b0

grouped[REJECTED].append(request)

The question remains if we want to have some other 4xx status code indicating that the job might run if it is resubmitted, but slivka's giving up trying to find somewhere for it to run at the moment.