Place to deinitialize tensor->extra after graph execution

[marty1885]

I'm working on a backend for Tenstorrent's devices. Due to their design, it's not always possible to get buffers to map to some pointer in the host address space (ex: the allocated memory lives on a chip connected via ethernet). This limitation prompted me to look into the SYCL split device backend and how they deal with the situation. They defer the actual allocation to init_tensor and return a ad-hoc base address to make GGML happy.

That works for some time. I got LLMs working and coherent. However, I quickly notice I run into memory leaking on device. GGML is calling init_tensor multiple times with brand new tensors (same name and base address, but tensor->extra is wiped, these are all tensors to communicate with the CPU backend as I'm still lacking some operator support). Causing my backend to allocate multiple times for the same tensor.

After looking at the ggml_backend_buffer_i and other structures, I don't see a place to deallocate tensor->extra to avoid the leak. I believe the SYCL split device backend would experience the same problem, though I do not have the setup to test it.

And a related question. Why is GGML trying to reinitialize the tensors? It doesn't make much sense. It is also awkward that GGML knows to reuse the base address, but extra is being reset.

[slaren]

You need to keep track of the tensor extras and free all of them when reset is called. Most likely you will want to keep a pool of available extras and reuse them to avoid expensive allocations when initializing tensors.

        // (optional) reset any internal state due to tensor initialization, such as tensor extras
        void         (*reset)        (ggml_backend_buffer_t buffer);

[marty1885]

I have looked into reset. But it doesn't tell me which tensors can be safely sent back to the pool. I assume it causes all the tensors, including the weights to be reset?

[slaren]

Only the tensors allocated in that buffer. This will only be called for compute buffers.

[marty1885]

Thank you!