LLaVA LLaVA LLaVA LLaVA

[mirek190]

Finally LLaVA under llamacpp !

If someone do not know LLaVA is for picture recognition and maybe for video in the furfure :D

[ianscrivener]

Hooray !!!

[ianscrivener]

Is there a plan to implement grammer for LLaVA? That will be very handy, powerful!

[damian0815]

i am taking a stab at llama-cpp-python binding support and then LMQL support. taking a leaf from the text-generation-webui book and will try to encode the image as a base64 blob that can be embedded in the prompt string eg <img src="data:image/jpeg;base64,{img_str}"> where img_str is a base64 encoding of the image (ref https://github.com/oobabooga/text-generation-webui/tree/main/extensions/multimodal).

i'm not sure if this work has already been started elsewhere, please lmk if this is double effort ha

[damian0815]

work on the above started here (draft state): #3613

[Jipok]

Any chance for support CogVLM?
It seems to be much superior to llava in terms of capabilities.

[mirek190]

WOW - made tests - that model is insane!! Better that GPT4 vision ..how is that even possible?

[ianscrivener]

Agree! CogVLM is amazing. I'd love to be able to run it on my MacOS M2 16Gb!!!

However.. the model files are around 28Gb each zipped. There are five (5) models... I am not sure how many models are required for CogVLM to run. I am wondering if the very large model size means that there is not a use case for "running CogLVM at the egde"... running on consumer grade hardware.

[mirek190]

is 28 GB because is not quantized - I cheeked already ;)
btw - who is playing with AI models having 16 GB ram ... lol

[ianscrivener]

Yes, quantization would substantially reduce the size...

16Gb RAM - me!
Checkout the llama.cpp Manifesto.. "inference at the edge"

[aiaicode]

Finally LLaVA under llamacpp !

If someone do not know LLaVA is for picture recognition and maybe for video in the furfure :D

Let's not forget, video is nothing but 30 pictures/second. So technically if you take input from user via voice / text and inference with image at that exact second of the video, you will get the reply, so you can talk to a video today if you have a high performance computer / new M2 Macbook Pro.

Knowing what the entire video on the other hand is slightly different. Inference every second of the video and combine the entire result and feed it back to an LLM and you should know what's happening in the video at each second of the video.