How to use low fidelity vision mode on OpenAI API?

[Jonseed]

The vision models on OpenAI's API will accept low or high fidelity image processing. Supposedly, if you send a 512x512 image, it should automatically use the low fidelity setting. But I haven't found that to be the case. It still seems to process 512x512 images using the high fidelity setting, and charges you 3x more in tokens. So to use the low setting we need explicitly set the detail parameter to low.

Here are the docs:

Low or high fidelity image understanding

By controlling the detail parameter, which has three options, low, high, or auto, you have control over how the model processes the image and generates its textual understanding. By default, the model will use the auto setting which will look at the image input size and decide if it should use the low or high setting.

low will enable the "low res" mode. The model will receive a low-res 512px x 512px version of the image, and represent the image with a budget of 85 tokens. This allows the API to return faster responses and consume fewer input tokens for use cases that do not require high detail.

high will enable "high res" mode, which first allows the model to first see the low res image (using 85 tokens) and then creates detailed crops using 170 tokens for each 512px x 512px tile.

How can we set the detail parameter in Plush? From the looks of it you have to set this parameter inside the messages array, it is not just another top level parameter. Could we add these parameters using the Add Parameters node, like this?

messages::content::image_url::detail::low

[glibsonoran]

No the add messages node has no access to the 'role: user' input dictionary which contains the nested image url/file reference, that's a whole different part of the processing procedure. I have it set to auto, in that I don't specify it and auto is the default.

I didn't read this as saying that 512x512 would automatically be rated low by the auto setting, it seemed to be unstated exactly what criteria would be used for making this determination. It sounded like it makes a low res 512x512 version of the image and additionally 512 x 512 crops using more tokens if it determines that 'high' would be the appropriate setting. It might even look at the prompt to determine if it can provide a reasonable response just using a low-res version of the image, I don't know.

Anyway, this is something that is going to be outside the scope of this project for the time being. You could experiment with the image and prompt and see if you could trigger a low res/low token response, like maybe just asking for the background color or something that doesn't require much resolution.

[Jonseed]

They say they "look at the image input size and decide if it should use the low or high setting." That sounds like the decision is based on the size of the input image. Unfortunately, 512x512 input images are still processed at the high setting for some reason. Maybe they do consider the prompt, like you say, and if needed process the image once at low res and then process it again in some other way for better comprehension. Anyway, thanks for considering it.

[EnragedAntelope]

Have you tried setting parameters? I just asked Copilot, which uses ChatGPT as backend, and it says for a local file that you're submitting via vision, this would be the syntax. You'd have to update it to go in the format of Plush Nodes parameters but it's probably worth a shot.

import openai

openai.api_key = 'your-api-key'

prompt = "Describe the image"

with open("path/to/your/image.png", "rb") as file:
    response = openai.Image.create(
        model="gpt-4-vision-preview",
        messages=[
            {"role": "user", "content": [{"type": "text", "text": prompt}]},
            {"role": "image_file", "image_file": {"file": file, "detail": "low"}}
        ],
        max_tokens=300
    )

print(response.choices[0].message['content']['text'])