Any way to automatically set the amount of GPU layers based on available GPU memory?

[samos123]

I don't want to have to experiment to find the optimal number for -ngl and rather have llama.cpp choose for me. So is there anything built in or anyone has a recommendation of how I could do this outside of llama.cpp?

[BarfingLemurs]

But with ggml, you should always offload all layers.

There's 4 model sizes.
If you used 4bit llama2 models, at the default 4k context, then there should only be 4 total memory values, where you can decide the appropriate model that fits.

It might also be difficult for developers to optimize if the nvidia drivers automatically offload to ram nowadays. :(

[samos123]

But with ggml, you should always offload all layers.

but that will cause an error like this:

CUDA error 2 at /llama-cpp-python/vendor/llama.cpp/ggml-cuda.cu:6524: out of memory

So I'm left with figuring out exactly how many of the 42 layers should go GPU. Right now it's trial and error for me.

[JohannesGaessler]

No.