How to read the VRAM usage on macOS?

[miku1958]

Is there any way to read VRAM usage on macOS? I tried the same model on macOS and windows, with metal turned on on macOS and cuBLAS turned on on windows, and -ngl to -1, and ended up using only 2G of RAM and a total of 13G of unity RAM on macOS, I'm assuming that it's 11G of VRAM being used here, and more than all of my RAM(16GB) and 13G of VRAM on windows.

Another question, is there a way to make windows use as much memory as macOS? I also try -ngl to 0 from #4310, but it is not helping

[easp]

On MacOS it hides in Wired Memory during inference and File Cache when idle.

[ggerganov]

You can use --no-mmap to get correct readings of the memory usage. Otherwise, the system tools don't reflect the correct values

[x4080]

In mac os is there any difference using =ngl 1 and - ngl 10 ? I'm using apple silicon

[crasm]

yes, just recently I think because I got a slowdown with ngl 1:

FINEST : 000172: LlamaCpp: llm_load_tensors: offloading 1 repeating layers to GPU
FINEST : 000172: LlamaCpp: llm_load_tensors: offloaded 1/81 layers to GPU
FINEST : 000172: LlamaCpp: llm_load_tensors:      Metal buffer size =   669.45 MiB
FINEST : 000172: LlamaCpp: llm_load_tensors:        CPU buffer size = 53965.19 MiB

[x4080]

Thanks, what is the optimum size for 16GB of memory ?

[psugihara]

hey @x4080, you ever figure this out (empirically or otherwise)?

[x4080]

@psugihara i just use ngl 99 I think

[psugihara]

Cool, that's what I'm trying too. Seems good 🤷