Understanding Job Failure - Memory Issues?

[jwpetley]

This is not a key priority and I am still trying to understand exactly how it happens.

I am currently running toil-cwl-runner to submit jobs on the spider cluster. This works very well for the pipeline and can greatly speed things up. However, quite often aoflagging jobs on the concatenated MSs fail. I have --retryCount=2 set, and often they complete on a retry when more memory is provided. If they completely fail then luckily with toil I can use the--restart flag and it will repair and continue running.

I thought I would make this an issue now so others can see it and comment their experience. It could be important for updating the job cpu and memory requirements as we move forwards.

I would say on a typical run I find 5/24 subbands fail and need to be restarted. Maybe @jurjen93 has some more experience with his ELAIS-N1 runs.

[tikk3r]

I've seen the aoflagging fail with someone else as well. In that case it was ultimately a resource issue in terms of how much memory it thought it could use vs how much was actually available with the amount of jobs on the nodes.

It currently defaults to 15%, but that basically means if more than 6 simultaneous jobs run on a node (which can relatively easliy happen nowadays with high core count nodes) it can could into problems. I think this step might need some smarter logic to deduce how much memory it is set to use. In my experience memory allocations are less common to be implemented on Slurm systems, which would've given an easier route to take.

[tikk3r]

A quick check on Spider and cosma does seem to imply that memory is at least used in the scheduling as I can't use crazy values. @jwpetley is it possible to check whether you've had more running on a node than was expected from a ~15% memory per job and/or if the reported memory usage was out of line with that?

[jwpetley]

I think it might be hard to get the details on what happened to a specific node but I can give an example of what the default submission appears to be:
Issued job 'CWLJob' concatenation.AOflagging.aoflagging kind-CWLJob/instance-owqek2vl v1 with job batch system ID: 31 and disk: 1.0 Gi, memory: 127.8 Gi, cores: 6, accelerators: [], preemptible: False

I'm not exactly sure how much memory is on the nodes but indeed could be problem if many land on the same

[jwpetley]

Sme info on currently running job:

[wfedfn-jpetley@ui-02 delay_cal]$ scontrol show jobid -dd 27130101
JobId=27130101 JobName=toil_job_5_concatenation.AOflagging.aoflagging
   UserId=wfedfn-jpetley(71942) GroupId=wfedfn-jpetley(71405) MCS_label=N/A
   Priority=14775 Nice=0 Account=wfedfn QOS=normal
   JobState=RUNNING Reason=None Dependency=(null)
   Requeue=0 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0
   DerivedExitCode=0:0
   RunTime=00:46:30 TimeLimit=12:00:00 TimeMin=N/A
   SubmitTime=2025-08-21T11:20:26 EligibleTime=2025-08-21T11:20:26
   AccrueTime=2025-08-21T11:20:26
   StartTime=2025-08-21T12:38:42 EndTime=2025-08-22T00:38:42 Deadline=N/A
   SuspendTime=None SecsPreSuspend=0 LastSchedEval=2025-08-21T12:38:42 Scheduler=Main
   Partition=normal AllocNode:Sid=wn-la-16:464875
   ReqNodeList=(null) ExcNodeList=(null)
   NodeList=wn-la-16
   BatchHost=wn-la-16
   NumNodes=1 NumCPUs=18 NumTasks=1 CPUs/Task=6 ReqB:S:C:T=0:0:*:*
   ReqTRES=cpu=6,mem=148655M,node=1,billing=6
   AllocTRES=cpu=18,mem=148655M,node=1,billing=18
   Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=*
   JOB_GRES=(null)
     Nodes=wn-la-16 CPU_IDs=47,64-80 Mem=148655 GRES=
   MinCPUsNode=19 MinMemoryNode=148655M MinTmpDiskNode=0
   Features=(null) DelayBoot=00:00:00
   OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null)
   Command=(null)
   WorkDir=/project/wfedfn/Data/L727112/delay_cal
   StdErr=/project/wfedfn/Data/L727112/delay_cal/workdir/toil_6a95fed7-c985-44d2-a110-e2e005dc6a09.5.27130101.err.log
   StdIn=/dev/null
   StdOut=/project/wfedfn/Data/L727112/delay_cal/workdir/toil_6a95fed7-c985-44d2-a110-e2e005dc6a09.5.27130101.out.log
   TresPerTask=cpu=6

[jwpetley]

And some current usage stats:

[wfedfn-jpetley@ui-02 delay_cal]$ sstat --format=AveCPU,MaxDiskRead,MaxDiskWrite,AvePages,AveRSS,JobID -j 27130101 --allsteps
    AveCPU  MaxDiskRead MaxDiskWrite   AvePages     AveRSS JobID        
---------- ------------ ------------ ---------- ---------- ------------ 
  00:00:00         7206        1.00M         14       144K 27130101.ex+ 
  02:19:22 196644280419 152387481257      10639 152039644K 27130101.ba+ 

[jwpetley]

Sometimes I get a warning like this:
WARNING: DP3 will use more memory than available. 1.55876e+11 bytes are available, but using 1.59617e+11

but these jobs do not necessarily fail

[lonbar]

Is there a straightforward method to see how much memory is required according to both toil and DP3? Perhaps they come up with different numbers, and jobs sometimes have less memory available than DP3 thinks.

For example, toil expects 127.8 GiB, while DP3 mentions 156 and 160 GB.