Fine-tune LLM with llama.cpp community data to enhance search efficiency

[bobqianic]

I've noticed that many questions within the llama.cpp community are closed once they're resolved. As more problems arise and are subsequently addressed, the number of closed questions continues to grow, making it increasingly time-consuming for newcomers to search through them. Furthermore, since these questions are very specific, they rarely appear in LLM's training data, resulting in answers of average quality. It would be beneficial if someone could develop a model specifically tailored for the llama.cpp community : )

[ianscrivener]

Yes, the absence of documentation for llama.cpp is a bit of an issue for many the users of llama.cpp; The same/similar questions are asked repeatedly in Discussion. Many readmes are empty. Much of the valuable information is buried in Git commit comments. Advanced concepts are not unpacked and explained. Advanced ML/LLM knowledge is somewhat presumed.

There seems to be a rush (arms race) to devleop LLM inference engine functionality... though explaining and documenting that for a broader audience doesn't happen so much.

That said - llama.cpp was created/intended as a C++ library/API that (IMO) is primarily/predominantly targeted/aimed at hardcore C++ developers. IMO it is not intended as a drop in "LLM inference engine" for a broader developer community

@ggerganov mentoned "I imagine llama.cpp mainly as an experimentation playground where we explore new strategies for LLM inference, we optimize things and we come up with fun ideas for new applications..."

Despite that - the 'example' apps like ./main and ./server are quite usable as-is... and the repo overall has ~5.7k forks and 40k++ stars. I'd guess that 99% of those people are not hardcore C++ developers

@ggerganov also mentioned that the project has attracted funding, and perhaps has Microsoft Azure support - but we are not seeing a move to a more commercial/mainstream approach like other LLM related opensource projects such as langchain, llamaindex, some of the vector databases etc - a pretty website, documentation and some specific 'products'. Perhaps that is in the pipeline? Perhaps llama.cpp will remain a leading and SOTA opensource C++ R&D project.

BTW: I have played around with RAG and a vector database - injesting the llama.cpp READMEs - then asking questions. But the initial results were quite poor. My thoughts were that manually extracting some commit comments would add to the knowledge "corpus".. a labourious taks. But my current view is that overall there isn't yet enough knowledge in the llama.cpp repo's READMEs + commit comments for there to be sufficient data for a useful RAG vector database or Lora to be created.