Is it possible to load llavar? Apparently it has superior text recognition to Llava

[IridiumMaster]

https://llavar.github.io/
I am a little unclear about how one might go about getting the projection for this, etc. Any help appreciated

[IridiumMaster]

I followed the steps as indicated in the examples/llava folder for model preparation, and I get the following error:
clip_model_load: model name: openai/clip-vit-large-patch14-336
clip_model_load: description: image encoder for LLaVA
clip_model_load: GGUF version: 3
clip_model_load: alignment: 32
clip_model_load: n_tensors: 375
clip_model_load: n_kv: 18
clip_model_load: ftype: f16

clip_model_load: text_encoder: 0
clip_model_load: vision_encoder: 1
clip_model_load: llava_projector: 1
clip_model_load: model size: 565.60 MB
clip_model_load: metadata size: 0.13 MB
libc++abi: terminating due to uncaught exception of type std::runtime_error: get_tensor: unable to find tensor mm.0.weight

Abort trap: 6

For anyone wanting to reproduce, the url I got the model from was:
https://huggingface.co/truehealth/LLaVar/tree/main

And the commands I ran included:

python3 ./examples/llava/llava-surgery.py -m models/Llavar
python3 ./examples/llava/convert-image-encoder-to-gguf.py -m models/clip-vit-large-patch14-336 --llava-projector models/Llavar/llava.projector --output-dir models/Llavar
python3 ./convert.py models/Llavar
./llava -m /models/Llavar/ggml-model-f16.gguf --mmproj /models/Llavar/mmproj-model-f16.gguf --image /images/tslacopy.jpg

I got the clip model from here:
https://huggingface.co/openai/clip-vit-large-patch14-336/tree/main

[monatis]

A quick look at its config.json reveals that it uses an older version of LLaVA architecture (not compatible with LLaVA 1.5). So it'll require extra work in C++ code in addition to the conversion. We can support it if it's really worth it --but is there a metric / benchmarking for a comparison between this one and newer LLaVA 1.5 checkpoints?

[IridiumMaster]

I don't see a ready comparison between this and the newer Llava 1.5, though in my tests Llava has almost no ability to deal with text in images well, so it might win by default. I will do a little research and get back to you on this. It is possible there is an alternative that is worthier. Thank you for your considerate and quick reply.