Semantic document search using dates to narrow down the search

[shayanalipour]

Hi, I want to perform a semantic search on +80 million news articles for which I have a publishing date. Before conducting the semantic search, I want to narrow down the news articles and only search within the given date range.

I managed to use Elasticsearch to do the semantic search of over 7 million news articles just by using an input text as a query and getting pretty solid results. However, for the whole dataset written in the document store, it doesn't make sense for me to do the search over the whole 80 million news articles when I know the date range of what I'm looking for!

For each row of my dataset, I have these features: Topic, URL, Date, Context

Which I turned the dataframe into Haystack's DocumentStore format, the result is a dictionary as below for each news article:

dict = {'content': news_df['context'], 'meta': {'datetime': news_df['Date'], 'url': news_df['URL'], 'topic': news_df['Topic']}}

I'm using

1. document_store = ElasticsearchDocumentStore(host="localhost", username="", password="", index="document")
2. retriever = BM25Retriever(document_store=document_store)
3. ready-made pipeline DocumentSearchPipeline

To be clear, I have a sentence with a date. I want to find the most relevant news articles given that sentence. Since I have the sentence and the news articles publishing date, I want to narrow down the news articles for the given date (a few days before and after maybe) and then conduct the semantic search.

Here's the example of using filters for BaseElasticsearchDocumentStore
But I can't find such a thing for ElasticsearchDocumentStore, also don't know how to put it together with the DocumentSearchPipeline.run() since these are the inputs for the method def run(self, query: str, params: Optional[dict] = None, debug: Optional[bool] = None

I can use search_fields in ElasticsearchDocumentStore and add the Date feature as one of the search fields to Retrieve that day's news articles but still don't know how to make it work for the semantic search in DocumentSearchPipeline. How to tell the pipeline to filter the news first based on the date range and then do the semantic search on the previous query results.

Thanks in advance for the time you will kindly dedicate to help me out with this issue.

P.S. I wanted to use FAISS but building the indexing for this size took forever. I came up with the below index based on this article guide
index = faiss.index_factory(768, "PCA64,IVF16384_HNSW32,Flat")

[julian-risch]

Hi @Squishy-33 thank you for explaining your setup in such detail. Your BM25Retriever can handle any filters via the filters parameter in its retrieve method:

haystack/haystack/nodes/retriever/sparse.py

Line 114 in a2905d0

filters: Optional[Dict[str, Union[Dict, List, str, int, float, bool]]] = None,

However, note that the search won't really be a semantic search but only a keyword-based search if you use ElasticsearchDocumentStore with a BM25Retriever. For semantic search you would need to use an EmbeddingRetriever and with your number of documents a DocumentStore other than ElasticsearchDocumentStoremakes more sense (faiss or milvus). The indexing takes much longer than, that's true even with a GPU. However, the indexing is needed only once and it is required for semantic search to calculate a vector representation for every document in your DocumentStore.

[shayanalipour]

Thanks @julian-risch for your help, I appreciate it.

I tried this and it worked!
filters = { "datetime": {"$gte": str_dt, "$lt": end_dt}}
pipe = DocumentSearchPipeline(retriever)
res = pipe.run(query['text], params={"Retriever": {"top_k": 10}, "filters": filters})

I first tried creating the datetime range according to this discussion but to no avail, so I went with the above-mentioned approach.

I used the DocumentSearchPipeline pipeline instead of calling the BM25Retriever's retriever function. However, I don't know what would be the advantage of using a pipeline here.

I will definitely try FAISS after being done scaling up ES on the whole dataset, just for the sake of comparing the results and performances.
Thanks again.

[julian-risch]

Great to hear it works for you now! 🙂
There is no big advantage of DocumentSearchPipeline over BM25Retriever. It's a layer of abstraction. Internally, DocumentSearchPipeline just uses:

        self.pipeline = Pipeline()
        self.pipeline.add_node(component=retriever, name="Retriever", inputs=["Query"])

and provides pipeline.run() without a need for the user to access the BM25Retriever node directly.