LLava Server

[jcgeo9]

Hello,

I have deployed a llava v1.5 server using the following command:
./server -c 4096 -ngl 100 -m ../ggml-model-q4_k.gguf --host ux-nic-8 --port 8013 --mmproj ../mmproj-model-f16.gguf

I am now trying to make an API call, like i normally do with other LLMs but I am having trouble formatting the prompt since it is a combination of image and text.

I have searched about this, but no luck. Is there anybody that can help me understand how to format both the image and the text in the prompt to feed it to the llm as following?

api_url="API URL"
headers = {"Content-Type": "application/json"}

pload = {
  'prompt': ***PROMPT***,
  'temperature': 0,
  'top_k': 40,
  'top_p': 0.95,
  'n_predict': 300,
  'stream': False,
  'repeat_penalty': 1.0,
  'ignore_eos': False}
rpload=json.dumps(pload)

response = requests.post(api_url, headers=headers, data=rpload)

[antonymevin]

api_url="API URL"
headers = {"Content-Type": "application/json"}

pload = {
  'prompt':  f"USER:[img-1]{prompt}\nASSISTANT:",
  'image_data': [{'data':image_data, 'id': 1}],
  'temperature': 0,
  'top_k': 40,
  'top_p': 0.95,
  'n_predict': 300,
  'stream': False,
  'repeat_penalty': 1.0,
  'ignore_eos': False}
rpload=json.dumps(pload)

response = requests.post(api_url, headers=headers, data=rpload)

You can add an image_data field in the payload and refer it in the prompt via image id.