How to specify which GPU to use in the config file

[bpben]

Hi all,
Is there a way to specify which gpu-id to use in the config file? I see in spacy there's an apply function which takes a gpu-id as an argument, but I couldn't track down how to do this with the assemble command as currently implemented in spacy-llm. Any advice here? What's the best practice for specifying device id?

[rmitsch]

Hi @bpben, in line with Hugging Face's transformers.pipeline you can include device=cuda:ID (e. g. device=cuda:0) in your models' config in config_init:

[components.llm.model]
@llm_models = "spacy.Dolly.v1"
name = "dolly-v2-3b"
config_init = {"device": "cuda:0"}

[bpben]

Thank you so much, was looking all over for this.

I'm guessing because spaCy doesn't support multi-gpu, there's not a clean way to use spacy-llm with multiple GPUs?

[rmitsch]

Thank you so much, was looking all over for this.

We should document this more clearly.

I'm guessing because spaCy doesn't support multi-gpu, there's not a clean way to use spacy-llm with multiple GPUs?

As of now*, all of the local LLMs supported by spacy-llm are wrapped Hugging Face transformers models and we are passing config_init and config_run on to these HF models. So limitations and capabilities of these models are not related to spaCy, but to the HF transformers library.

* Pretrained spaCy-native LLMs might be available soon thanks to our new curated-transformers library.