Why does decoding sometimes yield bunches of "\n" at end instead of EOS?

[dougdew64]

With n_len set to 1024 for the simple.cpp code, and a prompt of:

What is a large language model?

Decoding yields:

A large language model is a type of artificial intelligence (AI) model that is trained on a large corpus of text data to generate language outputs that are coherent and natural-sounding. These models are designed to learn the patterns and structures of language by exposure to a vast amount of text data, and they can be used for a variety of natural language processing tasks, such as language translation, text summarization, and language generation.

Which is followed by a whole lot of "\n" tokens.

Is there a simple explanation for why this happens?

Please pardon my naive question. I am new to this stuff.

[dougdew64]

In the Xcode console view (lower right) note that 1016 tokens were decoded. Most of those were "\n" at the end.

[dougdew64]

My prompt had 8 tokens. 1016 tokens were decoded. 8 + 1016 = 1024. I had n_len set to 1024. So, it seems that "\n" was decoded until n _len was reached.

And I've confirmed in the debugger that this line ends my decoding loop:

if (new_token_id == llama_token_eos(ctx) || n_cur == n_len) {

[dougdew64]

[dougdew64]

So, I never do receive an EOS token.

[dougdew64]

[KerfuffleV2]

A lot depends on the model, your sampling settings, whether or not the model is instruct tuned, whether or not you're actually using the correct prompt format for instruct-tuned models.

You didn't supply any details about the model you were using or whether you just prompted it with exactly just the question (usually won't work that well since it doesn't follow any known instruction format).

[dougdew64]

Thanks @KerfuffleV2.

I'm using llama-2-7b-chat.Q5_K_M.gguf.

I prompted with exactly just the question. I hadn't realized that I needed to do more.

I'm very new to this, and am attempting to ramp up by learning from this code base. Please pardon any naïveté on my part.

[KerfuffleV2]

No problem. It looks like the prompt format for vanilla llama2 chat is like:

[INST]What is a large language model?[/INST]

LLM stuff moves really fast and (in my opinion) that model is kind of outdated. This is what I'd suggest trying: https://huggingface.co/TheBloke/dolphin-2.2.1-mistral-7B-GGUF

There's information about the recommended prompt format on the page I linked. Here's an example with your question:

<|im_start|>system
You are Dolphin, a helpful AI assistant.<|im_end|>
<|im_start|>user
What is a large language model?<|im_end|>
<|im_start|>assistant

Scroll down near the bottom to see the original model creator's information (the stuff before that is added by TB who quantized the model to GGUF and includes mostly general information).

[dougdew64]

Thank you for your incredibly helpful answer @KerfuffleV2.

I will do as you recommended.

Have a great night!

[dougdew64]

@KerfuffleV2 you are easily my favorite person today.

Specifying the prompt with [INST]What is a large language model?[/INST] solved the problem which I had been experiencing.

I will switch to the newer model which you recommended and will read that model's instruction manual.

[dougdew64]