Why llama_build_graph() function is called for every token generation?

[Nick-infinity]

I am trying to understand the flow for forward pass / graph compute / inference .
Please correct me if my understanding is wrong.
The flow for inference starts after model and tensors are parsed from file.

llama_decode() -> llama_decode_internal() -> llama_build_graph() (creates llm structure) -> llm.build_llama() -> ggml_build_forward_expand() -> ggml_build_forward_impl() -> ggml_visit_parents() -> llm.free()

I think llama_decode() is called for every new token generation in decoding stage. This will call llama_build_graph() and all the nodes and tensor mapping to each node will happen every time for a new token generation.

1. Is my understanding correct?
2. Will it not add to the latency for new token. generation as this chain is called again and again?
3. Can we store the mapping of tensors and nodes in a "state" and just called "execute()" on nodes in-order?

@ggerganov

[ggerganov]

1. Yes
2. It will
3. The graphs for generating a sequence can be pre-built once and reused which would reduce the overhead from building the graph on-the-fly for each llama_decode(). However, this is not readily available through the existing API, though it can be achieved by hacking llama.cpp

There are many custom optimizations like this that can be applied based on the specific use case. With time, we will try to support these, but it takes time to arrive at the correct API

[Nick-infinity]

Thanks! I have few ideas to separate the preparation and execution in different stages to mininize overheads. Thanks for replying.

I am also trying to understand the architecture of newly introduced backend interfaces and implementation to come up with my custom backend. Do we have any documentation or UML yet?

[slaren]

There isn't any documentation at the moment, but it is very simple. At the most basic level, you have to provide functions for managing buffers and for evaluating graphs. I would suggest looking at the implementation of the CPU backend in ggml-backend.c to get started. Understanding the way graphs are represented and evaluated in ggml is more complicated, but you can use the implementations of any of the backends as an example.

[vgabbo]

Hi @Nick-infinity , did you try to implement this optimization? Noticed any improvement for inference-time? I'm trying to study the library llama.cpp, but my C/C++ is kind of rusty and it's taking time to understand how the flow goes...
Still, I was very curious about this idea. Thank you in advance.