TypeChat with llama.cpp

[abichinger]

I wanted to try TypeChat without an API key.

Luckily, llama-cpp-python already offers a web server which can be used as a drop-in replacement for the OpenAI API. So it was relatively easy to implement.

Result
TypeChat with Llama

[ChristianWeyer]

Did you try to use the Restaurant example with Llama2, e.g. 13B?
It gives me garbage (using llama-cpp-python 0.2.9)...

Any success on your side @abichinger ?

[abichinger]

Hi,

I just tried it. Seems like the 13B model can't handle the restaurant example.

Here is the output I got with max_tokens set to 4000:

🍕> I'll take a medium Pig with no arugula
{
  "items": [
    {
      "itemType": "pizza",
      "size": "medium",
      "addedToppings": [
        "pepperoni"
      ],
      "removedToppings": [],
      "quantity": 1,
      "name": "Pig In a Forest"
    }
  ]
}
    1 medium pizza with pepperoni, mushrooms, basil, Canadian bacon, arugula
🍕> I'd like two large, one with pepperoni and the other with extra sauce.  The pepperoni gets basil and the extra sauce gets Canadian bacon.  And add a whole salad. Make the Canadian bacon a medium. Make the salad a Greek with no red onions.  And give me two Mack and Jacks and a Sierra Nevada.  Oh, and add another salad with no red onions.
Response is not JSON:
 I'm sorry but I cannot understand your request as it contains multiple unsupported features such as "extra sauce", "Canadian bacon", "Greek salad" and "no red onions". Please try again using only supported features.
Please provide a valid JSON object with the supported features or clarify your request.

Have you tried the 70B model?

[ChristianWeyer]

I cannot run 70B locally and still need to test it against a cloud instance.