What are other lexical retriever which can be used other than BM25?

[amitguptadumka]

Since BM25 is not supported with ElasticSearch are there any alternative to it? I want to use a hybrid combination of dense search+Lexical/keyword search. Are there any options?

[dosubot[bot]]

🤖

Hello @amitguptadumka,

It's good to see you again. I hope you're doing well.

Regarding your question about alternative lexical retrievers to BM25, the LlamaIndex codebase does support a variety of other retrievers. Some of these include VectorIndexRetriever, SummaryIndexRetriever, KGTableRetriever, and many more. You can find the complete list in the __all__ list in the llama_index/retrievers/__init__.py file. Here is the link to the file: llama_index/retrievers/init.py

As for your question about ElasticSearch, I wasn't able to find specific information within the LlamaIndex repository about whether ElasticSearch supports any of the lexical retrievers currently implemented in the LlamaIndex codebase. However, ElasticSearch does have its own set of built-in scoring algorithms, which include TF/IDF, language models, and others. You might want to explore these options to see if they meet your needs.

If you're looking to use a hybrid combination of dense search and lexical/keyword search, you might want to consider using a multi-stage retrieval approach. In this approach, you could use a dense retriever (like a deep learning model) to retrieve a shortlist of candidate documents, and then re-rank these documents using a lexical retriever. This could potentially give you the best of both worlds.

I hope this helps! If you have any other questions or need further clarification, please don't hesitate to ask.

Best regards.

Sources

• llama_index/retrievers/init.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.