Do mistral don't support system prompt?

[TikaToka]

I am trying with interactive server, but it looks like it does not work

[cmp-nct]

Is there any LLM that supports a system prompt right now ? I don't think so.
Some come with one but they ignore changes to it mostly. In all LLMs I have tested you'll be better in doing the "system" prompt in the normal way of addressing it. that includes chatGPT/GPT4 which offer a system prompt that basically only ruins the output if you use it.

[KerfuffleV2]

I intentionally did not want to steer it in any direction with its responses, just informing it its name and the fact that it was an AI assistant.

I don't understand what you were testing or what the results are supposed to signify then. As I understand it, the point of the system prompt is to make the model do something.

It definitely doesn't surprise me that a system prompt with no purpose (and that in fact actually probably confuses the model) makes it perform worse.

also blanking out the system prompt had a similar significant negative impact, so the only conclusion is that the built in system prompt, whatever it is, is helping the model perform better.

It doesn't really work that way. There isn't a system prompt it will "assume" if you just leave the system prompt blank. Just for example, the Alpaca models that had "Below is an instruction that describes a task. Write a response that appropriately completes the request." as the "system prompt" (which wasn't meant to be configurable) still required writing exactly that. You could skip it and move directly to prompting the model, but it was generally recommended not to.

So again, it doesn't really surprise me that a system prompt that doesn't do anything to prepare the model for its role is going to perform worse than nothing on a model that was trained to use the system prompt. I mean, during training involving the system prompt, there would have been something actually meaningful in there.

[TikaToka]

@cmp-nct @alt8git I agree that default system prompt is optimized value for chat purpose, but I am utilizing the LLMs to do another job (not just chatting, I need specific format of output to define pernamently), so utilizing system prompt is needed for me :(

@KerfuffleV2 thanks! i'll try dolphin version ;)

[KerfuffleV2]

@alt8git

I assumed the guy that created the model used that system prompt for a reason.

You changed it though, and also "Dolphin2.1" with the words run together like that could potentially confuse the model even if it understood "Dolphin".

Anyway, being told it's an AI assistant could make it function better for assistant type stuff. A lot of people would their assistant to be personable, for example. This isn't something that would help with the Hellaswag test though, and in fact could be harmful.

thus confirming that either specifying or erasing system prompt with a blank can degrade model performance.

You really don't have to prove that and you definitely don't have to convince me. It absolutely is possible.

This conversation was in the context of the other person saying using a system prompt basically always noticeably decreases the quality of responses. I'm actually not even arguing the opposite of that: that there are definitely cases where using the system prompt is a benefit. I'm just saying we haven't proven that.

[TikaToka]

@KerfuffleV2 @alt8git I tried to use Dolphin, but there's an segmentation fault like

{"timestamp":1699425443,"level":"INFO","function":"main","line":2492,"message":"HTTP server listening","hostname":"127.0.0.1","port":9090}
all slots are idle and system prompt is empty, clear the KV cache
slot 0 is processing [task id: 0]
slot 0 : kv cache rm - [0, end)

print_timings: prompt eval time =     634.08 ms /   289 tokens (    2.19 ms per token,   455.78 tokens per second)
print_timings:        eval time =    3443.55 ms /   154 runs   (   22.36 ms per token,    44.72 tokens per second)
print_timings:       total time =    4077.64 ms
{"timestamp":1699425447,"level":"INFO","function":"log_server_request","line":2156,"message":"request","remote_addr":"127.0.0.1","remote_port":44238,"status":200,"method":"POST","path":"/completion","params":{}}
slot 0 released (444 tokens in cache)
updating system prompt
GGML_ASSERT: llama.cpp:5867: n_tokens <= n_batch
dolphin1.sh: line 2: 658919 Aborted                 (core dumped) CUDA_VISIBLE_DEVICES=6 ./server -m models/dolphin-2.2.1-mistral-7B-GGUF/dolphin-2.2.1-mistral-7b.Q5_K_M.gguf -c 2048 --port 9090 -ngl 35

I ran with

CUDA_VISIBLE_DEVICES=6 ./server -m models/dolphin-2.2.1-mistral-7B-GGUF/dolphin-2.2.1-mistral-7b.Q5_K_M.gguf -c 2048 --port 9090  -ngl 35

Do you know anything related to this error? before posting new issue or discussion, I wanted to ask you as I thought you might know this

[slaren]

The n_tokens <= n_batch assert fails when doing exactly that, an evaluation with more tokens than the batch size configured for the context. The evaluation needs to be split into multiple batches in that case, and if the server example doesn't do that, then that's probably a bug in the server example. However, we should probably change llama_decode to do that automatically for ease of use.