Can HuggingFace's utilities for generation be used with llama.cpp? #4139
-
|
Hello, llama.cpp developers. I would like to use HuggingFace's utilities for generation with llama.cpp. Can I use all of their utilities such as SequenceBiasLogitsProcessor? Thanks in advance! |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 1 reply
-
|
I think the short answer here is "no". That's meant for interfacing with their PyTorch-based stuff. It's not impossible that at some point you might be able to use it (someone would have to write some kind of interface, which wouldn't be a simple matter), but as far as I know there's currently no way. If I had to guess, I'd say your best bet is to wait and see if they do something to make it work with their Candle project which I believe can load GGUF files. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks, @KerfuffleV2! I will wait. |
Beta Was this translation helpful? Give feedback.
I think the short answer here is "no". That's meant for interfacing with their PyTorch-based stuff.
It's not impossible that at some point you might be able to use it (someone would have to write some kind of interface, which wouldn't be a simple matter), but as far as I know there's currently no way. If I had to guess, I'd say your best bet is to wait and see if they do something to make it work with their Candle project which I believe can load GGUF files.