Use BM25 retriever with haystack

[sinaayyy]

I use the FAISS document store and I would like to know if it is possible to use a BM25 type retriever?

[anakin87]

FAISSDocumentStore is based on the open-source library FAISS, which focuses on efficient similarity search of dense vectors.

To understand better the difference between vector-based (dense) and keyword-based (sparse) retrieval, you can read this section of the docs.

FAISSDocumentStore only support dense retrieval, so it is not compatible with BM25Retriever.

To understand better the different Document Stores and their compatibility with Retrievers, please have a look at this table.
(Please note that some Document Stores support both sparse and dense Retrievers.)

[sinaayyy]

Is the data deposited on ElasticSearch via haystack secure? suitable for a commercial setting?

[anakin87]

I think yes, but I also tag @masci who is more experienced than me about Elasticsearch...

[sinaayyy]

Okay, thanks, maybe he can give me more information

[nickprock]

FAISSDocumentStore is based on the open-source library FAISS, which focuses on efficient similarity search of dense vectors.

To understand better the difference between vector-based (dense) and keyword-based (sparse) retrieval, you can read this section of the docs.

FAISSDocumentStore only support dense retrieval, so it is not compatible with BM25Retriever.

To understand better the different Document Stores and their compatibility with Retrievers, please have a look at this table. (Please note that some Document Stores support both sparse and dense Retrievers.)

Hi @anakin87 , on the linked table I found that InMemoryDocumentStore doesn't support BM25, but in initialization function there is a parameter use_bm25=True. I'm confused

[anakin87]

Hey @nickprock! It is supported...
I recently opened an issue about this wrong statement: #4955