Why memory usage not change when add different input with GGML format

[SiraHaruethaipree]

I don't know too much about GGML format. But I know memory usage in vram GPU was changed depending on input sequence if input is long sequence it will increase the memory usage like when I test with load_8_bit or load_4_bit method from huggingface. So I need to know how memory usage is always the same value when use GGML format with GPU. Please someone explain.
Thanks

[KerfuffleV2]

I believe it just pre-allocates memory based on the max context size you set (-c). So it won't really matter too much how much of that your prompt consumes.

[SiraHaruethaipree]

Thanks, As you said. After I try to change n_ctx, It will pre-allocate memory based on context size.

[cmp-nct]

There is barely any memory allocation going on post load, that's why loading takes a bit.
The parameters -c -ngl are mostly responsible how memory is allocated. Batch processing (-b default 512) also causes memory consumption.

In addition you have a bit of dynamic memory consumption for example when using cublas which internally runs on 16 or 32 bit, maybe fp8 support will be added in some time to improve that. The type of model influences that a lot of course, bigger tensors need more temporary conversion memory.

You can control that behavior, best dig into the options available at -h, some things are controlled at compile time.