are cluster limits different for kbatch jobs than for interactive ones?

[erthward]

Hi, Tom and crew!

I am currently running an analysis job using kbatch job submit. I try to use a cluster consisting of 50 16-GB, 8-core workers, but I get the message GatewayWarning: Scale request of 50 workers would exceed resource limit of 13 workers. Scaling to 13 instead.. I didn't see a 13-worker limit expressed anywhere in the docs, and I don't see any math that explains how the 'Cluster Limits' defined here work out to a 13-worker limit in my particular case.

Is there something I'm misunderstanding, or are the cluster limits somehow different for kbatch jobs?

Thanks!

Drew

[jessjaco]

As another user, I've noticed that those cluster limits seem to be obsolete. I've only been able to scale to 100 8GB workers (rather than 400) for the last few months. When I bump to 16GB the limits seem to be randomly smaller (recently 56). I think they're adjusting the quota and haven't updated the documentation. I think it's possible it may also depend on the current usage across the entire "computer".

[erthward]

Thanks for the insight! That makes a lot of sense.

@ MSPC folks: It will be important to me to figure out how exactly the limits are currently working, if possible. This is because I initially decided to move a project onto MSPC in order to be able to easily scale a Monte Carlo analysis without needing to do a lot of DevOps work. If I now need to move the analysis elsewhere then that will have an impact on the project timeline that I'll need to communicate out right away.

Thanks again for all of your hard work! It's a fantastic tool.

[TomAugspurger]

Thanks for the report and sorry for the delay.

I think the latest cluster limits are defined at https://github.com/microsoft/planetary-computer-hub/blob/4668f37d4d987ab1c7a2c5bd2f403700b9936f00/helm/chart/config.yaml#L319-L321, but I might be wrong. I haven't had any time to work on kbatch-related stuff for a while now.

[erthward]

Thanks, Tom! That helped a lot. I realized that the limit I was running into was because I was trying to use too many cores overall. And now I have a good idea of where to look for this info going forward, too.