Hosting. ColQwen/Colpali

[simjak]

@adithya-s-k Thanks for handy library. I am also experimenting with vision rag by following Vespa's solution.
https://pyvespa.readthedocs.io/en/latest/examples/pdf-retrieval-with-ColQwen2-vlm_Vespa-cloud.html

I have two questions:

• where do you host ColPali/ColQwen model ( I was running in RunPod)
• which vector database do you use?

[adithya-s-k]

@simjak

I would recommend the following from my personal use:

1. Hosting:
   Use EmbedAnything to inference Colpali/Colqwen on Runpod. This library acts as an inference layer on top of these models. It’s well-written and actively maintained.

2. Storage for Retrieval:
   Handling retrieval can be tricky since these models use 2D embeddings instead of traditional one-dimensional embeddings. Here are some options:

   • Vespa Engine: Offers extensive customizability for different types of indexing.
   • LanceDB: Great for local testing and experimentation.
   • Qdrant: Supports 2D embeddings and has optimizations for Colpali. While I haven’t personally used it, this blog post might be helpful.

A good full stack application that i have seen with vision rag is here - repo

[simjak]

Thanks! This is useful!
I'm planing to use Vespa, for VPS maybe Hetzner, for GPU maybe RunPod or Baseten or Hetzner.
Planing to self host Vespa in Hetzner