Is it possible to return log probs?

[wmpauli]

I have been trying to return log probs so I can implement something along the lines of VQA Score (https://arxiv.org/abs/2404.01291), but am having the impression that this is currently not supported.

I focused on Gemini-1.5-flash, because I know that the model supports it, Unfortunately, the model only supports it if used with the method "models.generateContent" [https://ai.google.dev/api/generate-content], but not with "models.streamGenerateContent", which is the one used by ell. I confirmed this by monkey-patching the path (path = "publishers/google/models/gemini-1.5-flash:generateContent vs. streamGenerateContent) to use the non-streaming version. However, with that change translate_from_provider method of course fails (expected).