semantic scores of all documents for a query

[sinchanabhat]

Hello,

I am trying to get the semantic scores of all the documents for a single query. Is there any variable that I need to pass in while using

pipeline = DocumentSearchPipeline(retriever)
result = pipeline.run(query, params={"Retriever": {"top_k" : 10}})

If I use or do not use top_k, I will get ordered scores, but I do not want ordered scores, I want scores for all the documents. Is this possible?

Looking forward!
Thanks
Sinchana

[sjrl]

Hi, @sinchanabhat I converted this to a discussion since this looks more like a question about how to use existing Haystack functionality.

It would be easiest for me to help you if you could provide me with runnable code to reproduce your setup. So if you could provide that to me I would really appreciate it.

But to take a crack at answering your question:

1. I would first recommend to make sure you are using an EmbeddingRetriever (docs for using this node are here) since the models used by that node will provide you with semantic scores.
2. You can access the scores of the returned documents by accessing them through the result. For example result['documents'][0].score will give you the score for the first document. The score of the next document would be result['documents'][1].score and so forth.
3. If the issue is that you only get the scores for the top_k amount of documents and you would like the scores for all documents in your document store. I would recommend increasing the top_k value to equal the number of documents in the document store. Just be aware that returning all documents like this can be very slow depending on how many documents you have in your document store.

len_doc_store = doc_store.get_document_count()
result = pipeline.run(query, params={"Retriever": {"top_k" : len_doc_store}})

Please let me know if this helps!