Auto calculate max number of layers with -ngl -1?

[chigkim]

If the entire model and context size don't fit, can we have a feature -ngl -1 where that automatically calculates the maximum number of layers that could fit into the gpu, and offload the rest to cpu?

[Jeximo]

a feature -ngl -1 where that automatically calculates the maximum number of layers that could fit into the gpu, and offload the rest to cpu?

It's an important feature, but cannot be done without very significant changes to llama.cpp, and backend interfaces to handle this feature correctly.

[fredlas]

Since it sounds like the clean in-code approach of #6502 is not imminent, I have written a script for this; please give it a try @WilliamTambellini and anyone else interested!

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.