Do the OAI completion endpoints automatically fix the prompt to have the correct format?

[codinalot]

For example, if I start the server with this:

./server -m ../llama/llama-2-13b/ggml-model-f16.gguf

Then I hit it from python using this:

completion = self.client.chat.completions.create(
            model="gpt-3.5-turbo-0125",
            response_format={ "type": "json_object" },
            messages=[
                {"role": "system", "content": system_prompt},
                {"role": "user", "content": text}
            ]
            )

I can't tell if I need to change my user content's text to match the llama-2-13b model I'm running's prompt format from their paper, which is the following format:

<s>[INST] <<sys>> systemprompthere <</sys>> prompt [/INST]

Any help is appreciated!

P.S.- is there a way to require the server output json? I couldn't find a flag for using a json grammar with the server.

[phymbert]

Hello, the server is printing the chat template at startup.

You can set --verbose for verbose logs in addition with --log-format text.

[codinalot]

The chat-template is different than the prompt-template though, no?

[codinalot]

I'm also not sure if this answers my question as to whether I need to be formatting my prompts before inputting them. It actually raises a separate question, which is whether the following format will work natively with the server mode:

messages=[
                {"role": "system", "content": system_prompt},
                {"role": "user", "content": text}
            ]

I'm unable to find documentation about this particular nuance, but if you know where it is please let me know!

[ngxson]

AFAIK, chat template is used for chat model (system-user-assistant-user-assistant...) and prompt template is used for instruction model (only prompt-response).

Using chat endpoint may not be the best choice if you're not using a chat model. I propose using /completion endpoint instead of /chat/completions, which allow you to format the prompt yourself and server only does the completion part.