Is the QnA piplne.run work always with only 10K documents?

[Kandaraa]

Hello

I'm currently discovering your tool and it's just awesome! I'm just wondering if the method below retrieve only 10K documents:
prediction = pipe.run( query="my question?", params={"Retriever": {"top_k": 10 }, "Reader": {"top_k": 10}} )

Because when I activate the debug mode I can see the Elasticsearch respond and if figure out like this

DEBUG - elasticsearch - < {"took":155,"timed_out":false,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"hits":{"total":{"value":10000,"relation":"gte"},"max_score":13.804194...........

Noting that my Elasticsearch index contain 2M documents. I don't know if I'm doing it right or not. Thus, if there is another way to work with the 2M documents at the same time please don't hesitate to clarify it.

Thank you

[anakin87]

Hello @Kandaraa! Let's try to clarify your doubts...

1. I'm not an Elasticsearch expert, but I can say that "hits":{"total":{"value":10000,"relation":"gte"} means that more than 10K documents from your whole index match your query. See this Stackoverflow thread if you want to understand better.

2. Using your Haystack Question Answering pipeline (retriever + reader), the following operations are performed:

   • the retriever finds the top 10 documents in your whole Elasticsearch index: these are the documents that are most relevant for your query. The retriever directly calls Elasticsearch and saves the top 10 results.
   • starting from these documents, the reader extracts the top 10 answers to your question.

Read more about the retriever and the reader in the Haystack Docs.