Possible for a MIG_STRATEGY=mixed configuration?

[AlienResidents]

Hi,

I see in v1.34 MIG has been introduced. Does v1.34 only support a MIG strategy of "single", or can I somehow use "mixed"?

Referencing the k8s-device-plugin, but I can't see comparable options in v1.34.
https://github.com/NVIDIA/k8s-device-plugin?tab=readme-ov-file#configuration-option-details

[KCSesh]

Thanks @AlienResidents for reaching out and opening this discussion.

As of now, the MIG feature provided on Bottlerocket starting at v1.34.0 - only supports "single" mode meaning that the GPU(s) can be partitioned into multiple instances with the same partition sizes on the node.

I would be interested to hear more about your use-case if you are interested in using "mixed".

[AlienResidents]

Thanks @KCSesh

I would be interested to hear more about your use-case if you are interested in using "mixed".

Our use case is for AWS EKS, and a p4d.24xlarge running a single GPU workload using vLLM on GPUs 0-3, and 28 GPU workloads on GPUs 4-7 with a 1g.5gb config. This is to minimise cost. Happy to provide some more details.

[AlienResidents]

Thanks @KCSesh

I would be interested to hear more about your use-case if you are interested in using "mixed".

Our use case is for AWS EKS, and a p4d.24xlarge running a single GPU workload using vLLM on GPUs 0-3, and 28 GPU workloads on GPUs 4-7 with a 1g.5gb config. This is to minimise cost. Happy to provide some more details.

Also note, this is only during peak periods. Off-peak we use another solution. We use KEDA + Karpenter to automate off-peak workloads.

[KCSesh]

@AlienResidents Are you running multiple p4d.24xlarge instances?

Could you elaborate on your preference to use MIG with "mixed" instead of a set of p4d.24xlarge instances that are not using MIG, and another set of p4d.24xlarge that is with a 1g.5gb config ?

[AlienResidents]

@KCSesh We are a startup (cost savings is a focus) running a single p4d.24xlarge instance (~$31k AUD/month), and 28 g4dn.xlarge (~$14k AUD/month) instances. All of our GPU workloads (29 LLMs) can fit on a single p4d.24xlarge instance, if we can get "mixed" working with Bottlerocket. Currently we're considering switching to an Amazon Linux AMIs, but we'd rather stick to Bottlerocket for our LLM AMIs as that is the rest of our Fleet, and especially for the xvdb volume snapshot pre-loading of container images.

Currently 1 LLM (via vLLM) requires the GPU memory of a p4d.24xlarge (GPUs 0-3), so we don't have choice for that LLM, and our smaller 28 LLMs are currently running on 28 g4dn.xlarge nodes, but we want to leverage the additional 4 GPUs on the p4d.24xlarge for cost savings. We use KEDA + Karpenter currently for autoscaling, and we can get rid of that complexity and cost by utilising a single p4d.24xlarge using a "mixed" MIG strategy.

[AlienResidents]

@KCSesh Any thoughts on my use case above? Does my use case warrant a "mixed" MIG_STRATEGY feature? I can try to provide more details if you'd like.

[KCSesh]

@AlienResidents thanks for following up. I have opened a feature request here: #4483

Please feel free to provide any details you think is important.