When posting a request to the server, what is the format for specifying logit biases?

[phasiclabs]

Does anyone know what the format for specifying logit biases? I can't find any info on it!

I've been posting json like this to the lllama.cpp server, and everything works apart from the 'logit-bias'
{
'prompt': "",
'n_predict': 128,
'temperature': 0.9,
'mirostat': 2,
'mirostat-ent': 5,
'image_data': [{'data': '', 'id': 12}],
'stream': True,
'logit-bias': [[, 100], [, 100], [, 100]]
}

From looking at the server code, it seems to expect an array of arrays, but when I breakpoint it it find the logit-bias array but can't handle the values inside, so does nothing.

[KerfuffleV2]

I checked the code. It looks like it want an array of token id/bias pairs or instead of bias you can use a bool and false is the same as setting negative infinity as the bias. So that would be [[TOKEN_ID, BIAS], [TOKEN_ID, BIAS]] or more specifically [[123,1.1], [345,-1.0]].

[phasiclabs]

Thanks - that is actually the format I was trying to make work - my original post looked like that, but had the token id stripped out by Github for some reason, because I put them inside less than/greater than symbols.

I'll try debugging it again, but it seems to find the outer array, but then then detects inner arrays as 'objects' rather than arrays, so this test fails :

if (el.is_array() && el.size() == 2 && el[0].is_number_integer())

[phasiclabs]

Turns out it works fine in you use 'logit_bias' (as documented), and not 'logit-bias' as I had been using!