Is there a benefit to use ROCm on AMD instead CPU with a little VRAM ?

[grigio]

I did some tests but it seems the performance do not change much. I have a little VRAM on my integrated APU

[KerfuffleV2]

Probably not. APUs don't even have dedicated VRAM, they're just using normal system RAM. It's worth trying out, but a very possible result is that just using CPU is better. This also can be the case for old GPUs even though they have dedicated VRAM.

Note that there can be differences in prompt processing vs generation so it could be worth using GPU just for prompt ingestion and CPU for generation. (i.e. compile with the GPU support but use -ngl 0). When I had my old Nvidia 1060 it was much faster than CPU for prompt input/running perplexity but considerably slower for actually running inference.

[grigio]

I've an APU, Ryzen 7 7700 with a little of VRAM but it segfaults #2797