Support for Llama-2-7B-32K-Instruct?

[quarterturn]

https://huggingface.co/togethercomputer/Llama-2-7B-32K-Instruct

"Model Description

Llama-2-7B-32K-Instruct is an open-source, long-context chat model finetuned from Llama-2-7B-32K, over high-quality instruction and chat data."

[KerfuffleV2]

Doesn't look like it needs anything special. You might need to set rope scaling.

[MichaelDays]

Was tinkering with this model yesterday, trying to get it to summarise some text. Not a lot of success, but my instruct promptfu is weak. :)

[quarterturn]

works well with llama-cpp-python and llama.cpp. I was able to use this on a single 3090:

llm = Llama(model_path="/home/qtr/Documents/llama.cpp/models/llama2-7b-32k-instruct-ggml-f16/llama2-7b-32k-f16", n_gpu_layers=35, rope_freq_base=1047620, rope_freq_scale=0.25, n_ctx=16386)

I was unable to get the model to work properly at anthing other than f16, though. Even 8_0 resulted in broken replies.