is it possible to add logprob entry to server?

[ghost]

Server doesn't have a logprob function such as https://textsynth.com/documentation.html#logprob. This function can be useful in evaluation harnesses such as MMLU to find what the model would pick for a multiple choice questions or other types of yes/no or limited choice questions. I don't know enough about the codebase now if it would be easy or hard to add this but it could be useful if possible.

[GKeppler]

Is this implemented already? I would need it for the same purpose of QA like in the Harness Implementation of MML. Or can you recommend a different paython package for fast LLM inference?