Move llama grammar parser into llama.cpp

[spirobel]

can we please move the grammar parser by @ejones into llama.cpp and expose it with a C compatible API?

There is no way to use this feature currently when importing libllama.so, because it is behind a cpp namespace.

maybe we can change / add a new API like this:

    LLAMA_API struct parse_state * llama_parse_grammar(const char * src);

    LLAMA_API struct llama_grammar * llama_grammar_init_from_parsed(struct parse_state * parsed_grammar);

what do you think?

[ggerganov]

Yes, we should do that. I was even considering moving it to ggml since we are going to also use it in whisper.cpp, but not sure how difficult it would be to reimplement in C. For now we can merge it in llama.cpp and later reconsider

[spirobel]

Another option would be to move it into a separate library. I am exploring this path at the moment. llama.cpp is already getting big so it might be better to write some smaller example libraries.

Currently we only have libllama.so to link to. So that is why I went down this path at first. But then I just added another example and a new .so target in the make file. It would be great to see more .so example targets similar to the experiments in the example directory. So that is what I did.

I also saw that you started to group some common functionality in common.cpp like the sampling function for example:
https://github.com/ggerganov/llama.cpp/blob/99115f3fa654b593099c6719ad30e3f54ce231e1/common/common.cpp#L866

maybe common is the right place?

[ggerganov]

Creating a separate library is not desired - things will become too fragmented. common is a place mainly for reducing code-duplication in the examples. It contains functionality that is often useful and needed, but can be also implemented in various ways, depending on the language and project that want to use llama.cpp

[paschembri]

GPT-4 gave up so quickly I kind of surprised 😅

Converting the given C++ code to ANSI C would require a significant amount of work and is beyond the scope of this AI model.

[paschembri]

Would it make sense to use llvm ?