Finetune detection, new Instruction modes and "system prompt"

[cmp-nct]

I've noticed that the OpenAssist fine tune appears to use system commands in the training data for Falcon, it's not totally reliable but interesting. The new parameters will be available within the next hours.
"-enc" This uses fine-tune detection and encloses your prompt with the correct syntax for OpenAssistant, Wizard, Falcon Instruct and generic models
"-sys" This adds a system command to modulate the model into different behavior.
Command: -enc -sys "You now are a monster" -p "Write a poem"

Here is Falcon 40B (at just 2 bit quantization on single GPU, 23 tokens/sec for that poem):

I am a monster,
A creature of the night.
My eyes burn red with fury,
And my teeth gleam in the light.

I stalk my prey through the shadows,
Waiting for the perfect moment to strike.
I feast on the flesh of the innocent,
And I revel in their cries.

I am a monster,
A fiend from the netherworld.
My strength is unmatched,
And my hunger insatiable.

But there's one thing you should know:
Despite my terrifying form,
I am but a figment of your imagination,
A mere shadow in the storm.

Here is 5 bit Falcon 7B (55 tokens/sec for that poem):

I am a monster, big and tall
With sharp teeth and claws that can't be tamed
I lurk in the shadows, waiting to strike
And when I do, no one will survive.

My eyes glow red, like burning coals
My skin is rough, my muscles are cold
I am a creature of the night, a beast of prey
Who only seeks to feast on human prey.

But don't be afraid, little one
For I may look scary, but I'm not done
I can be kind, if you just show me love
And together we can conquer this world above.

[maddes8cht]

The description of the sys parameter sounds like how I always imagined the function of --prompt together with --keep -1 in Llama.cpp (and always felt that it didn`t really work well)

Therefore again explicitly for clarification:
--system is actually what shows up in Chatgpt playground as "system message":
It is a message to the system that always stays in the focus of the context and is permanently respected, even if other parts of a long conversation does get out of context.

Did i get this right?

The description in the helptext (falcon-main --help) is:

-sys, --system prefix the entire prompt with the system prompt text

I feel this is misleading insofar as it at least allows the interpretation that every single prompt message (or every message in an interactive session) is prefixed with the system-message - which would consume an enormous amount of context.
This interpretation is not completely absurd, because various browser frontends like Oobabooga and Kobold.cpp do exactly such things regularly.
As help text, i would therefore suggest something like

-sys --system   prepend a system message to the prompt or whole conversation that never runs out of context.

or

-sys --system   precedes the prompt or the entire conversation with a system message that never goes out of context.

(it's actually really hard to put this into a short description that doesn't get misleading..)

[cmp-nct]

I was not able to fully complete the system prompt yesterday. If possible I'll do it today.
But that's the general idea of a system prompt.
However in my current implementation it's not "raw", it is using special tokens that were tested to work with the model you use.
I suppose a "raw" method could be added alongside. Also those parts are not completely finished, it worked quite good with open assistant. But how good is seed based.
It's not that different with turbo-3.5, it might or might not follow the prompt

In general I am not sure if the current (same as llama.cpp just with the new additions) instruction loop actually works optimally, I found some strange states during debugging. It might need a fix or a rewrite.

In case you use that frequently, how do you use it ?

[maddes8cht]

"In case you use that frequently, how do you use it ?"
I would like to use it more frequently.
My use cases would be:
Provide a context.
Sometimes i like to discuss a longer conversation with some aspects that should always be kept in mind.
I may even ask a llm to provide a dense summary for this.

Sometimes i try storywriting with an llm helper.
Providing main information on the Plot would be helpful as elsewise it will loose focus and halucinate things that will not fit into the intended storyline.

Having this "system-message" thing more reliable would open lots of possibilities to fine-control the model.

When it comes to coding, (which I'm not really good at), how about:
Giving the system message the "interface-section" of a program, and asking it to give me the functions one by one.

Okay, starcoder and its descendants may be better at this than falcon.
But would be nice to get a system message there too.

[maddes8cht]

The main documentation of each "example" in Llama.cpp was intended to be in the readme.md file of the respective example folder (which is not very consistently done there).
Also the co-forked readme.md in falcon-main is somewhat outdated in the original Llama main example.
Nevertheless, I think that the main documentation should be located exactly there - possibly also with a prominent reference to it in the readme.md of the main page.

Would you like me to provide a revision of this readme.md in a PR?
In fact, I've started doing just that and am currently working in the new cli parameters.
I have corrected all mentions of Llama to Falcon (of course with a mention of the origin of the whole project of Llama.cpp)
But there is still some work to do until this is (again) a decent documentation, in particular I have some small questions if I got it right in each case.
Also, there are times when I could write, but I don't have access to a computer on which Falcon would run even halfway decent (only an ancient 2nd gen i7 with 8 GB ram with intel graphics) and therefore can't try out questions that come up myself.

[cmp-nct]

I actually did not notice that readme before, there is still a lot of not relevant code anymore.
My original idea was a ggllm.cpp - something that runs llama alongside falcon. But the amount of work required to unify them and to keep updating llama_main with the new falcon features is way too much.

Basically the main readme covers most of it at the moment, the only reason to use ggllm.cpp is to run Falcon models.
If you'd like to write examples into a example specific readme for main/perplexity/quantizer that's certainly welcome.
I just removed the totally outdated one.

Regarding computer usage:
You could go to one of the large cheap GPU-server hosters, some of them offer 100$ trial if you say you want to evaluate them. It would be a free way to start and 100$ allows to run a A10 or small A100 type GPU for many continuous days.

[cmp-nct]

Update:
I think the best place is the Wiki for actual helpful content. Not saying a custom readme is useless, but the wiki might be much better for separating it into topics.
https://github.com/cmp-nct/ggllm.cpp/wiki