MultiDB support

[rieger-jared]

Hey @Davidyz, as an attempt to see if I can resolve this issue I'd like to give integrating Qdrant and Pgvector a go; Would you be open to the possibility of supporting multiple DBs. I'm eager to have a crack at this myself so long as you'd be open to it.

I understand there would be quite some refactoring work which I don't mind to do at all. I think it shouldn't be too bad given that the project isn't too big right now. I've had a solid poke around the code and think it's possible to do within a reasonable amount of time.

I'm wanting to try this, as I haven't been able to solve the querying for large codebases which is a large blocker for me. I'm wondering if this is not an issue with different db types hence the suggestion.

What do you think?

[Davidyz]

Yes, if you're willing to give it a try, I'm definitely open to the PR. Feel free to open a draft PR as you work through it so that we can discuss changes before it's too late.

Also, since I'm not very familiar with the vectordb space, would you say that the alternative DBs can provide all functionalities that chroma provides? For example, the embedding model integration and the metadata feature, which some features are built on top of?

[Davidyz]

Also, from the project-management perspective, it might be easier for me if you split this into multiple PRs. For example, you could implement a wrapper class (like a class DBConnectorBase(ABC) which databases can inherit from) for chromadb and finish the refactoring in one PR, and introduce new DB connector classes in other PRs.

[rieger-jared]

I think that's a great idea. First focusing on having chroma work smoothly and then focusing on implementing more DBs sounds like a solid plan.