Skip to content

FIM/Infill changes vs. model support & GGUF regeneration etc.? #6708

Answered by ggerganov
ghchris2021 asked this question in Q&A
Discussion options

You must be logged in to vote

Is there a summary of status of which models are presently supported for the FIM use case given the recent changes?

All models that during conversion write special FIM tokens to the GGUF header should be supported:

https://github.com/ggerganov/llama.cpp/blob/599ce84a71512b72bf4fd6a248e7725f646eb1a8/convert-hf-to-gguf.py#L1306-L1312

https://github.com/ggerganov/llama.cpp/blob/599ce84a71512b72bf4fd6a248e7725f646eb1a8/llama.cpp#L4278-L4281

If support for a new model is needed, the convert scripts need to be updated accrodingly.

Deepseek models are still WIP: #5981

Replies: 1 comment

Comment options

You must be logged in to vote
0 replies
Answer selected by ghchris2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Category
Q&A
Labels
None yet
2 participants