How to allow the model to stop at eos token while using grammars?

[aalyousfi]

When I send the prompt below without grammars to a model served with a Llama.cpp server, the model ends the response with <|im_end|><dummy32000> and stopped_eos is true in the response.

However, when I send the same prompt with the JSON grammar, it ends the response with hundreds of newlines (\ns) and stopped_eos come as false and stopped_limit as true in the response.

So how can I preserve the model's ability to end the response when it actually has nothing more to say? In other words, how to make it able to stop when it reaches special tokens (like the eos token) while using grammars?

The prompt:

<|im_end|>\n<|im_start|>user\nWhat are the largest countries? Respond in JSON format.<|im_end|>\n<|im_start|>assistant

Data sent to the server:

data = {
        'prompt': prompt,
        'n_predict': 1024,
        'repeat_penalty': 1.1,
        'temperature': 0.1,
    }

and 'grammar': grammar is added of course when using grammars.

Thanks in advance!

[nuchi]

The JSON grammar includes whitespace since that's valid JSON. if you don't want whitespace after the returned object, you could remove the whitespace from the end of the json.gbnf object ::= definition. If you want whitespace internally but not at the end of the root object, you could do:

-root   ::= object
+root   ::= object-no-ws
 value  ::= object | array | string | number | ("true" | "false" | "null") ws
 
-object ::=
+object-no-ws ::=
   "{" ws (
             string ":" ws value
     ("," ws string ":" ws value)*
-  )? "}" ws
+  )? "}"
+
+object ::= object-no-ws ws
 
 array  ::=
   "[" ws (
             value
     ("," ws value)*
   )? "]" ws
 
 string ::=
   "\"" (
     [^"\\\x7F\x00-\x1F] |
     "\\" (["\\/bfnrt] | "u" [0-9a-fA-F] [0-9a-fA-F] [0-9a-fA-F] [0-9a-fA-F]) # escapes
   )* "\"" ws
 
 number ::= ("-"? ([0-9] | [1-9] [0-9]*)) ("." [0-9]+)? ([eE] [-+]? [0-9]+)? ws
 
 # Optional space: by convention, applied in this grammar after literal chars when allowed
 ws ::= ([ \t\n] ws)?