FIM/Infill changes vs. model support & GGUF regeneration etc.? #6708
-
I noticed the recent release notes about it and the following commit activity wrt. FIM / Infill and supporting more models e.g. CodeGemma having that capability but having various vocabulary symbols required to solicit it i.e.: I haven't scrutinized the details but I have a couple questions: Codellama (and I suppose its close derivative relations) was AFAICT the nominally originally supported model for Is there a summary of status of which models are presently supported for the FIM use case given the recent changes? And then I'm less sure about others e.g. the deepseek-coder variant below which mentions in its model card that https://huggingface.co/deepseek-ai/deepseek-coder-6.7b-base Also after b2680 changed the gguf vocabulary is my conception correct that if I've previously downloaded other models Thanks! It's nice to see codegemma etc. getting support for this, it was on my list to try for IDE use. Context: gguf : add special tokens metadata for FIM/Infill (#6689) This commit adds special token metadata for Fill-In-the-Middle The motivation for this is that currently there is support for CodeLlama |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment
-
All models that during conversion write special FIM tokens to the GGUF header should be supported: If support for a new model is needed, the Deepseek models are still WIP: #5981 |
Beta Was this translation helpful? Give feedback.
All models that during conversion write special FIM tokens to the GGUF header should be supported:
https://github.com/ggerganov/llama.cpp/blob/599ce84a71512b72bf4fd6a248e7725f646eb1a8/convert-hf-to-gguf.py#L1306-L1312
https://github.com/ggerganov/llama.cpp/blob/599ce84a71512b72bf4fd6a248e7725f646eb1a8/llama.cpp#L4278-L4281
If support for a new model is needed, the
convert
scripts need to be updated accrodingly.Deepseek models are still WIP: #5981