Get Specific logits

[micsthepick]

I'd like to compare two specific token probs, and ban all other tokens. (In batches)
One of the ways that I'm attempting to do so is with the following python request data:

    data = {
        "prompt": [...],
        "temperature": 0,
        "top_k": -1,
        "top_p": 1,
        "min_p": 0,
        "repeat_penalty": 1,
        "n_predict": 1,
        "logit_bias": [[str(i),False] for i in range(256*256) if i not in [yes_token_id, no_token_id]],
        "n_probs": 2,
        "add_bos_token": True,
        "use_samplers": True
    }
    ```
    I also tried `[[str(i), True] for i in range(256*256)]`, but in both cases it returns unrelated tokens, and also is much slower for the first case.

[micsthepick]

note, I can do the same with text-generation-webui with the following code:

AUTH = os.getenv("OPENAI_AI_KEY", testing_key)
testing_api = "http://127.0.0.1:5000"
api = os.getenv("OPENAI_API_ENDPOINT", testing_api)
route = "v1/internal/logits"
URL = f"{api}/{route}"

    data = {
        "prompt": (
f"""[INST]Determine whether the Bible text is applicable for answering the provided question[/INST]
[QUESTION]{question}[/QUESTION]
[TEXT]{hunk}[/TEXT]
Answer (Must be 'yes' or 'no' without quotes):"""
        ),
        "custom_token_bans": ','.join(str(i) for i in range(256*256) if i not in [5081, 708]),
        "top_logits": 2,
        "add_bos_token": True,
        "use_samplers": True
    }

    try:
        # Send POST request
        response = requests.post(URL, headers=headers, json=data, verify=False)
        response.raise_for_status()
    except requests.exceptions.RequestException as e:
        print(f"Request failed: {e}")
        sys.exit(1)

[micsthepick]

the following appears to work!

data = {
        "prompt": [await get_data(*value[:2]) for value in values],
        "temperature": -1,
        "n_predict": 1,
        "logit_bias": [[i,False] for i in range(256*256) if i not in [yes_token_id, no_token_id]],
        "n_probs": 2,
        "add_bos_token": True,
        "samplers": []
}