Support for Phi-2

[mlinegar]

Is it possible to use Microsoft's Phi models with LMQL, or are there any plans to make this possible? It seems to have been incorporated into llama.cpp (see here for example).

When I try to load the model, e.g. with:

lmql serve-model --loader auto-gptq --cuda --static /local/path/phi-2-GPTQ --disable_exllamav2 True --use_safetensors True --inject_fused_attention False --inject_fused_mlp False --trust_remote_code True

I get the error: phi-msft isn't supported yet.

Are there any plans to support this class of model?

[lbeurerkellner]

I have not tested phi-2 tokenizers with LMQL yet, but the error you seeing does not come from the LMQL runtime. It sounds more like the installed auto-gptq version does not yet support phi-2.