server consistently yields bad answers, yet same prompt to ./main consistently produces correct answer, is server broken or am I using it wrong?

[segmond]

started API with (I have tried with --chat-template and without the option)
$ ../llama.cpp/server --embeddings -c 8000 -ngl 81 --host 192.168.1.100 --port 8080 -np 4 -cb -c 4096 -b 4096 -ub 2048 -m ./mistral-7b-instruct-v0.2.Q8_0.gguf --chat-template llama2

message prompt to API ->
[{'role': 'system', 'content': 'You are a very smart and helpful AI assistant that answers questions in a logical manner. Your goal is to answer the given question using the sources you have been provided. Answer the question by providing one paragraph. Do you not mention the sources in your response. Do not number or list the answers in bullet form. Think carefully about your answer.'}, {'role': 'user', 'content': '\n Answer the question below using the sources to get relevant information. \n If the source doesn't provide information to answer the question, output "I don't know".\n Sources: Source 1: phenomenon as a whole without being accused of any of the specific heresies it sought to suppress. Another way to counterattack is with metaphor. Arthur Miller undermined the House Un-American Activities Committee by writing a play, "The Crucible," about the Salem witch trials. He never referred directly to the committee and so gave them no way to reply. What could HUAC do, defend the Salem witch trials? And yet Miller's metaphor stuck so well that to this day the activities of the committee are often described as a "witch-hunt." Best of all, probably, is humor. Zealots, whatever their cause, invariably lack a sense of humor. They can't reply in kind to jokes. They're as unhappy on the territory of humor as a mounted knight on a skating rink. Victorian prudishness, for example, seems to have been defeated mainly by treating it as a joke. Likewise its reincarnation as political correctness. "I am glad that I managed to write 'The Crucible,'" Arthur Miller wrote, "but looking back I have often wished I'd had the temperament to do an absurd comedy, which is what the situation deserved." 17 ABQ A Dutch friend says I should use Holland as an example of a tolerant society. It's true they have a long tradition of comparative open-mindedness. For centuries the low countries were the place to go to say things you couldn't say anywhere else, and this helped to make the region a center of scholarship and industry (which have been closely tied for longer than most people realize). Descartes, though claimed by the French, did much of his thinking in Holland. And yet, I wonder. The Dutch seem to live their lives up to their necks in rules and regulations. There's so much you can't do there; is there really nothing you can't say?\nSource 2: by a hack that all his contemporaries could tolerate that he felt there must be a better solution. Intolerance for ugliness is not in itself enough. You have to understand a field well before you develop a good nose for what needs fixing. You have to do your homework. But as you become expert in a field, you'll start to hear little voices saying, What a hack! There must be a better way. Don't ignore those voices. Cultivate them. The recipe for great work is: very exacting taste, plus the ability to gratify it. Notes Sullivan actually said "form ever follows function," but I think the usual misquotation is closer to what modernist architects meant. Stephen G. Brush, "Why was Relativity Accepted?" _Phys. Perspect. 1 (1999) 184-214. Japanese Translation Chinese Translation Slovenian Translation German Translation Interview: Milton Glaser Russian Translation * * * You'll find this essay and 14 others in Hackers & Painters.\nSource 3: I'm not sure if this is true. On one hand, it seems surprisingly difficult to waste your time so long as you're working hard on something interesting. So much of what you do ends up being useful. But on the other hand, the rule about the relationship between risk and reward is so powerful that it seems to hold wherever risk occurs. Newton's case, at least, suggests that the risk/reward rule holds here. He's famous for one particular obsession of his that turned out to be unprecedentedly fruitful: using math to describe the world. But he had two other obsessions, alchemy and theology, that seem to have been complete wastes of time. He ended up net ahead. His bet on what we now call physics paid off so well that it more than compensated for the other two. But were the other two necessary, in the sense that he had to take big risks to make such big discoveries? I don't know. \nSource 4: Is the mathematician a small man because he's discontented? No; he's just doing a kind of work that wasn't very common in Confucius's day. \nSource 5: If these guys were able to do what they did only because of some magic Shakespeareness or Einsteinness, then it's not our fault if we can't do something as good. I'm not saying there's no such thing as genius. But if you're trying to choose between two theories and one gives you an excuse for being lazy, the other one is probably right. So far we've cut the Standard Graduation Speech down from "don't give up on your dreams" to "what someone else can do, you can do." But it needs to be cut still further. There is some variation in natural ability. Most people overestimate its role, but it does exist. If I were talking to a guy four feet tall whose ambition was to play in the NBA, I'd feel pretty stupid saying, you can do anything if you really try. 2 We need to cut the Standard Graduation Speech down to, "what someone else with your abilities can do, you can do; and don't underestimate your abilities." But as so often happens, the closer you get to the truth, the messier your sentence gets. We've taken a nice, neat (but wrong) slogan, and churned it up like a mud puddle. It doesn't make a very good speech anymore. But worse still, it doesn't tell you what to do anymore. Someone with your abilities? What are your abilities? Upwind I think the solution is to work in the other direction. Instead of working back from a goal, work forward from promising situations. This is what most successful people actually do anyway. In the graduation-speech approach, you decide where you want to be in twenty years, and then ask: what should I do now to get there? I propose instead that you don't commit to anything in the future,\n\n Question: What did Arthur Miller wish he had the temperament for?\n Result:'}]

Terrible Result/Answer

Arthur Miller wished he had the temperament for a more straightforward and less introspective approach to life. In his essay "The Price of a Ticket," he expresses his admiration for people who can easily let go of their regrets and move on, while he found himself constantly dwelling on the past and revisiting his mistakes. He writes, "I have often wondered what it would be like to be the kind of man who, once having made a decision, carries it out without looking back, who does not brood over alternative possibilities and 'what ifs' but faces the future with a calm and resolute certainty."

However, it's important to note that Miller did not view this as a deficiency, but rather an aspect of his personality that fueled his creativity as a writer. He recognized that his introspective nature allowed him to explore the depths of human emotions and create complex and compelling characters. In the end, Miller embraced his temperament and used it to his advantage in his work.

---

Now, we try with main, I consistently get a correct result.

Message prompt formatted and ran through main gives a reasonable answer

$ ~/llama.cpp/main -c 4096 -m ./mistral-7b-instruct-v0.2.Q8_0.gguf -ngl 50 --log-disable -f zmi

ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: CUDA_USE_TENSOR_CORES: yes
ggml_cuda_init: found 5 CUDA devices:
Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
Device 1: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
Device 2: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
Device 3: Tesla P40, compute capability 6.1, VMM: yes
Device 4: Tesla P40, compute capability 6.1, VMM: yes
[INST] Answer the question below using the sources to get relevant information. If the source doesn't provide information to answer the question, output "I don't know". Sources: Source 1: phenomenon as a whole without being accused of any of the specific heresies it sought to suppress. Another way to counterattack is with metaphor. Arthur Miller undermined the House Un-American Activities Committee by writing a play, "The Crucible," about the Salem witch trials. He never referred directly to the committee and so gave them no way to reply. What could HUAC do, defend the Salem witch trials? And yet Miller's metaphor stuck so well that to this day the activities of the committee are often described as a "witch-hunt." Best of all, probably, is humor. Zealots, whatever their cause, invariably lack a sense of humor. They can't reply in kind to jokes. They're as unhappy on the territory of humor as a mounted knight on a skating rink. Victorian prudishness, for example, seems to have been defeated mainly by treating it as a joke. Likewise its reincarnation as political correctness**. "I am glad that I managed to write 'The Crucible,'" Arthur Miller wrote, "but looking back I have often wished I'd had the temperament to do an absurd comedy, which is what the situation deserved."** 17 ABQ A Dutch friend says I should use Holland as an example of a tolerant society. It's true they have a long tradition of comparative open-mindedness. For centuries the low countries were the place to go to say things you couldn't say anywhere else, and this helped to make the region a center of scholarship and industry (which have been closely tied for longer than most people realize). Descartes, though claimed by the French, did much of his thinking in Holland. And yet, I wonder. The Dutch seem to live their lives up to their necks in rules and regulations. There's so much you can't do there; is there really nothing you can't say?Source 2: by a hack that all his contemporaries could tolerate that he felt there must be a better solution. Intolerance for ugliness is not in itself enough. You have to understand a field well before you develop a good nose for what needs fixing. You have to do your homework. But as you become expert in a field, you'll start to hear little voices saying, What a hack! There must be a better way. Don't ignore those voices. Cultivate them. The recipe for great work is: very exacting taste, plus the ability to gratify it. Notes Sullivan actually said "form ever follows function," but I think the usual misquotation is closer to what modernist architects meant. Stephen G. Brush, "Why was Relativity Accepted?" _Phys. Perspect. 1 (1999) 184-214. Japanese Translation Chinese Translation Slovenian Translation German Translation Interview: Milton Glaser Russian Translation * * * You'll find this essay and 14 others in Hackers & Painters.Source 3: I'm not sure if this is true. On one hand, it seems surprisingly difficult to waste your time so long as you're working hard on something interesting. So much of what you do ends up being useful. But on the other hand, the rule about the relationship between risk and reward is so powerful that it seems to hold wherever risk occurs. Newton's case, at least, suggests that the risk/reward rule holds here. He's famous for one particular obsession of his that turned out to be unprecedentedly fruitful: using math to describe the world. But he had two other obsessions, alchemy and theology, that seem to have been complete wastes of time. He ended up net ahead. His bet on what we now call physics paid off s

What did Arthur Miller wish he had the temperament for? [/INST] Arthur Miller wished he had the temperament to write an absurd comedy instead of "The Crucible."

What might be the cause of this? How do I get the prompt for server to deliver the same results? I'm using openai API chat completion method.

[segmond]

I'm convinced server is broken. I take the same prompt that I pass to CLI, instead of going through the API I use server UI and have tried completion and chat and get the same bad result as API. Works great via CLI tho.

Here's the prompt

[INST] Answer the question below using the sources to get relevant information. If the source doesn't provide information to answer the question, output "I don't know". Sources: Source 1: phenomenon as a whole without being accused of any of the specific heresies it sought to suppress. Another way to counterattack is with metaphor. Arthur Miller undermined the House Un-American Activities Committee by writing a play, "The Crucible," about the Salem witch trials. He never referred directly to the committee and so gave them no way to reply. What could HUAC do, defend the Salem witch trials? And yet Miller's metaphor stuck so well that to this day the activities of the committee are often described as a "witch-hunt." Best of all, probably, is humor. Zealots, whatever their cause, invariably lack a sense of humor. They can't reply in kind to jokes. They're as unhappy on the territory of humor as a mounted knight on a skating rink. Victorian prudishness, for example, seems to have been defeated mainly by treating it as a joke. Likewise its reincarnation as political correctness. "I am glad that I managed to write 'The Crucible,'" Arthur Miller wrote, "but looking back I have often wished I'd had the temperament to do an absurd comedy, which is what the situation deserved." 17 ABQ A Dutch friend says I should use Holland as an example of a tolerant society. It's true they have a long tradition of comparative open-mindedness. For centuries the low countries were the place to go to say things you couldn't say anywhere else, and this helped to make the region a center of scholarship and industry (which have been closely tied for longer than most people realize). Descartes, though claimed by the French, did much of his thinking in Holland. And yet, I wonder. The Dutch seem to live their lives up to their necks in rules and regulations. There's so much you can't do there; is there really nothing you can't say?Source 2: by a hack that all his contemporaries could tolerate that he felt there must be a better solution. Intolerance for ugliness is not in itself enough. You have to understand a field well before you develop a good nose for what needs fixing. You have to do your homework. But as you become expert in a field, you'll start to hear little voices saying, What a hack! There must be a better way. Don't ignore those voices. Cultivate them. The recipe for great work is: very exacting taste, plus the ability to gratify it. Notes Sullivan actually said "form ever follows function," but I think the usual misquotation is closer to what modernist architects meant. Stephen G. Brush, "Why was Relativity Accepted?" _Phys. Perspect. 1 (1999) 184-214. Japanese Translation Chinese Translation Slovenian Translation German Translation Interview: Milton Glaser Russian Translation * * * You'll find this essay and 14 others in Hackers & Painters.Source 3: I'm not sure if this is true. On one hand, it seems surprisingly difficult to waste your time so long as you're working hard on something interesting. So much of what you do ends up being useful. But on the other hand, the rule about the relationship between risk and reward is so powerful that it seems to hold wherever risk occurs. Newton's case, at least, suggests that the risk/reward rule holds here. He's famous for one particular obsession of his that turned out to be unprecedentedly fruitful: using math to describe the world. But he had two other obsessions, alchemy and theology, that seem to have been complete wastes of time. He ended up net ahead. His bet on what we now call physics paid off s

What did Arthur Miller wish he had the temperament for? [/INST]

[segmond]

So I tried server using openai API, tried it using the UI, and then for the heck of it tried it using curl. It still fails in all 3 ways

script I used to test via curl

PROMPT=cat prompt.txt
PROMPT=$(printf '%s\n' "$PROMPT" | sed 's/"/\"/g')
PROMPT_ESCAPED=$(echo "$PROMPT" | sed ':a;N;$!ba;s/\n/\n/g' | iconv -t ASCII//TRANSLIT//IGNORE)
curl --request POST
--url http://192.168.1.100:8080/completion
--header "Content-Type: application/json"
--data "{"prompt": "$PROMPT_ESCAPED","n_predict": 128}"

[ggerganov]

Using -c 4096 -np 4 will result in just 4096/4 = 1024 tokens of context window which seems not enough to fit this prompt. You can either try to reduce -np or increase -c. For example, try -c 4096 -np 1

[segmond]

Woops, sorry about that, and thanks for the response!