is there some way to set the main cuda device?

[bryanhpchiang]

llama_model_load_internal: using CUDA for GPU acceleration
ggml_cuda_set_main_device: using device 0 (NVIDIA GeForce RTX 3090) as main device
llama_model_load_internal: mem required  = 2135.98 MB (+ 1608.00 MB per state)

right now, one of the devices gets automatically selected but is there a way to set this manually via CLI args?

[SlyEcho]

It is documented in the README

But basically it is supposed to be controlled by the environment variable CUDA_VISIBLE_DEVICES.

[mongolu]

I've used this variant, setting CUDA_VISIBLE_DEVICES=1,0 but still
ggml_cuda_set_main_device: using device 0
Can't it be defined as an input parameter?
In llama.cpp line 875 it's:
/.main_gpu =/ 0,
Doesn't this means it's hardcoded to be device id 0?

[SlyEcho]

if you set CUDA_VISIBLE_DEVICES=1,0 then both 1 and 0 are visible. The number 0 inside llama.cpp means the first visible. If you don't want device 0 then don't include it like so: CUDA_VISIBLE_DEVICES=1

[mongolu]

The situation is that I want all GPU's to be used, but main gpu to be one other than id 0.
How can I do that?

[jsemrau]

On Ubuntu, I set the environment variable like this
export CUDA_VISIBLE_DEVICES=4,0,1,2,3
Making my device "4" the main gpu

[mongolu]

From the link above it shoud respected the order in CUDA_VISIBLE_DEVICES, but it's not happening.
Only the devices whose index is present in the sequence are visible to CUDA applications and they are enumerated in the order of the sequence. If one of the indices is invalid, only the devices whose index precedes the invalid index are visible to CUDA applications. For example, setting CUDA_VISIBLE_DEVICES to 2,1 causes device 0 to be invisible and device 2 to be enumerated before device 1.

[mongolu]

Hi, could be implemented?

[SlyEcho]

There is also the option -mg i, --main-gpu i the GPU to use for scratch and small tensors now.

[mongolu]

Thank you for lighting this up for me.