What data format should I use for ggml-vocab-llama.gguf ?

[GermanAizek]

Interested opportunity to train model so that example was like this.

Test train data:

#QUESTION
5 + 5
#QUESTION

#ANSWER
10
#ANSWER

#QUESTION
-1 - 10
#QUESTION

#ANSWER
-11
#ANSWER

Test chat example:

== Running in interactive mode. ==
 - Press Ctrl+C to interject at any time.
 - Press Return to return control to LLaMa.
 - To return control without starting a new line, end your input with '/'.
 - If you want to submit another line, end your input with '\'.

User: 5+5########10
User: 5 + 5########10
User: 5 - 10#########
User: 8-1 - 10########-11
User: -1 -(10#######-11
User: 100 + 100#######

Answers should be strict, but question can be posed with extra characters or spaces.

[KerfuffleV2]

This sounds similar to #3917 - those aren't real models, they're for running the vocab tests. So they only include that type of model's vocabulary, none of the actual tensors.

[GermanAizek]

@KerfuffleV2, can you then advise me which open source project to take as basis for training?

[KerfuffleV2]

Actually, I was partially wrong. It seems like those models can be used as part of train: https://github.com/ggerganov/llama.cpp/tree/master/examples/train-text-from-scratch

However note that it has --vocab-model there. I'm not sure what you were actually doing - whether you were trying to train from scratch, fine tune an existing model, etc. Your best bet is to look at the README in examples like that sparse as it is.

There's also an example for fine-tuning: https://github.com/ggerganov/llama.cpp/tree/master/examples/finetune

Neither of those work by talking to the model interactively though, and it wouldn't be practical given the amount of data required to train a model.

[GermanAizek]

@KerfuffleV2, do I understand correctly that I need to retrain based on large model with addition my own data?

[KerfuffleV2]

Like I said, I'm not sure what you're trying to do and you didn't clarify so it's hard to answer that. Training a model from scratch takes a lot of resources though, so I'm going to guess what you probably want to do is fine-tune an existing model.

I'm not completely sure what you mean by "retrain", if you mean "fine-tune" then probably yes.

I haven't done any model training stuff so I can't really help you more than this.