Documentation of llama 2 model architecture details?

[dougdew64]

Where is the llama 2 model architecture documentation which was used to figure out what code to write for the llama.cpp file?

This llama doc does not seem to provide the kind of details which were apparently important when writing the code in the llama.cpp file: https://arxiv.org/pdf/2302.13971.pdf

My goal is to gain a thorough understanding of the code in the llama.cpp file.

Thank you in advance.

[dougdew64]

This llama doc also does not seem to provide all of the architectural details necessary to write the code in the llama.cpp file: https://arxiv.org/pdf/2307.09288.pdf

[slaren]

That's mostly correct, the graphs are for the most part predefined in the code. This is the function that creates the graph for llama models:
https://github.com/ggerganov/llama.cpp/blob/6961c4bd0b5176e10ab03b35394f1e9eab761792/llama.cpp#L3091
llama.cpp supports other models as well, and each of them have a similar llm_build function to create the computation graph.

Note that ggml already has an utility to export a visualization of a graph in graphviz dot file format. You can create it by uncommenting these lines, but you may also want to restrict the graph building code to just 1 layer, otherwise the graph will be huge.
https://github.com/ggerganov/llama.cpp/blob/6961c4bd0b5176e10ab03b35394f1e9eab761792/llama.cpp#L6027-L6030

[dougdew64]

Also, please pardon any of my misuses of terminology. I am new to this, and so am still very ignorant. For example, while reading a comment in the ggml.h file just now, I noticed the phrase "computation graph." So, perhaps my earlier use of "model graph" should have instead been "computation graph."

[dougdew64]

Thank you VERY much for pointing out that vizualization export function. You have been super helpful @slaren!

[dougdew64]

I used that graphviz export function to generate a llama.dot file. Then, I started the dot command to generate a llama.png file from the llama.dot file. So far, the dot command has been running for two hours at 100% cpu. Your advice to limit to just one layer is looking better by the hour. ;-)

[dougdew64]

Thirty hours of cpu time, and still no result from graphviz dot. It's probably fair to say that attempting to use dot to generate a *.png file from the export of a complete graph is a poor idea.