What's an embeddings model I can use right now?

[Johnhersh]

I'm having a hard time on the embedding part of a RAG app. It seems llama.cpp supports generating embeddings, but I get really poor results with any gguf model I try. I tried some Bert-based models but those don't load because Bert isn't supported, and I see some discussions about adding those but as of now nothing like that is available.

So what model do people use to generate embeddings right now?

[marcingomulkiewicz]

I've tried that with openchat 3.5, results were usually acceptable.

[Johnhersh]

Thanks for the tip, I tried this just now but I still get very poor result. Doing a cosine search on the embeddings generated from this one seems to yield irrelevant results.

[marcingomulkiewicz]

I wrote some relatively simple PoC, I tried something like that:

• PostgreSQL with PGVector as DB
• cutting text into fragments starting and finishing on a sentence boundary, with significant overlaps between fragments (like 5-10 sentences)
• openchat for both embedding and response generation (lame, I know)
• upon retrieval I'm merging consecutive fragments, removing the aforementioned overlaps

Tested it on CS6 Bible and Dune by Frank Herbert - but I must admit, the tests were not too extensive.

[marcingomulkiewicz]

@Johnhersh Not sure if it's of any help, but I noticed that when I use 'generate' method during response (I'm using llama.cpp through java-llama) I'm generally getting gibberish, but when I use 'complete' method the results are much more sensible - I can't explain this, but I've checked a number of times - the embeddings are eactly the same. ('generate' returns generated content piece by piece, while 'complete' collects the entire response before returning it) YMMV.

[SpaceCowboy850]

This one works for me so long as you don't need the GPU to work (CPU only).
https://github.com/xyzhang626/embeddings.cpp

It also supports BGE, which is one of the higher ranked hugging face embedding models.

[Johnhersh]

I unfortunately cannot get this working at all. I get a vector full of zeroes. Which is odd because with llama.cpp I usually see this if I run the server without the --embeddings option. But in that project that parameter seems missing?

Feels like I'm missing something very basic

[Johnhersh]

I guess I don't know how to use it. It doesn't seem to have the same server endpoints as llama.cpp has. Is it only meant for streaming?