Support llguidance #774
Support llguidance #774
Conversation
|
Is there a second PR required to expose the guidance APIs? i.e. what would it take to modify simple to guide output? |
There was a problem hiding this comment.
Looking into this, I can see why this would be a very cool feature.
However, I don't think this approach will do what you're expecting.
The only place the llguidance API's are exposed is in common/sampling.h, which we do not generate bindings to. I do not want this to change as keeping up to date with the main API is difficult enough as is with as little maintenance effort as this project currently receives.
On the bright side, llguidance is a rust library and I think a llama-cpp-rs + llguidance library would be pretty easy to build and maintain. Building a rust library (llguidance) as part of a c++ library (llama.cpp) to be used by a rust library (llama-cpp-rs) strikes me as the wrong approach.
Unless I'm mistaken I will probably not merge this PR.
|
Agreed. Thanks for the input. |
Resolves #701