tutorials : list for llama.cpp

[ggerganov]

Project: ggml-org : tutorials

List:

• tutorial : compute embeddings using llama.cpp
• tutorial : parallel inference using Hugging Face dedicated endpoints
• tutorial : KV cache reuse with llama-server
• tutorial : measuring time to first token (TTFT) and time between tokens (TBT)

TODO:

• Is there a way to cache multiple prompt prefixes? #13488
• How to use function calls? #13134
• how to measure time to first token (TTFT) and time between tokens (TBT) #13251
• Apple A-chipsets; how to estimate a suitable model size ? #12742
• How to get started with webui development (ref: tutorials : list for llama.cpp #13523 (comment))
• etc.

Simply search for "How to" in the Discussions: https://github.com/ggml-org/llama.cpp/discussions?discussions_q=is%3Aopen+How+to

Contributions for writing tutorials are welcome!

[ngxson]

Re. the UI stuff, as explained on this comment, I think frontend development in llama.cpp is no longer a beginner-friendly task as it used to be. The problem is that frontend development can be very hard to get it right (too many approaches to do the same thing.) Nowadays, I only accept PRs that follow modern web dev practices, especially practices on ReactJS and Typescript.

However, I think tutorial to document "llama.cpp-specific" setups, like how to build and bundle it with the binary, can be useful, as this is not common frontend dev knowledge.

[ggerganov]

Ok, I removed the TODO. Btw feel free to edit directly the OP. I just added a few examples for potential tutorials to get things going.

[smahs]

On prompt caching

[immaixq]

I’d like to work on the tutorial section on measuring Time to First Token (TTFT) and Time Between Tokens (TBT) in llama.cpp as answered in this discussion. Would appreciate feedback on the approach, any preferred format under show and tell discussion.

[ggerganov]

@immaixq There isn't any strict format for the tutorials - whatever makes sense to you. We will discuss the specifics in the discussion itself.