KeyError : 'Retriever' while implementing Evaluation example: https://haystack.deepset.ai/tutorials/05_evaluation

[tosingithub]

Please, I need help to resolve the error I'm getting while trying to replicate the example in the mentioned Evaluation guide. When it gets to the retriever_result = eval_result["Retriever"], I get the following error. The same error occurs for the Reader, if I comment out the earlier line. I'm using FAISSDocumentStore instead of ElasticSearch but everything else is essentially the same.

    retriever_result = eval_result["Retriever"]
  File "/home/oluade/e_ir/lib/python3.8/site-packages/haystack/schema.py", line 824, in __getitem__
    return self.node_results.__getitem__(key)
KeyError: 'Retriever' 

[masci]

@tosingithub the tutorial in Colab works for me, it's hard to say what's not working in your local environment. Feel free to add more details but otherwise we can't help with this little context you provide.

[tosingithub]

Yes @masci, the Colab works. Thanks. The problem arises when I change the DocumentStore to FAISSDocumentStore. Can you please provide the same or a similar example that works for FAISSDocumentStore? Thanks

[silvanocerza]

Those are two different DocumentStores with different API and inner working. It's not that easy to switch from one to another and shouldn't be done lightly.
I suggest you give a read to the DocumentStores documentation to better understand their differences.

Also I suggest you take a look at other tutorials that use FAISS like the Better Retrieval with Embedding Retrieval. That might give you more insights on how to use other DocumentStores.

[tosingithub]

Ok @silvanocerza but I would like an example for evaluation that is based on FAISSDocumentStore. Kindly help with this. Thanks

[silvanocerza]

I'm deeply sorry but we really don't have the throughput to provide ad hoc examples to everyone. Also this is no small example and we don't even have the whole context to provide a satisfying solution.

I already provided a tutorial that uses FAISS that might be helpful, and the related FAISSDocumentStore documentation. That should be enough of a starting point to understand how to fix your issue.

I would suggest to ask over in the Haystack Discord, there's always people from the community willing to help there. You can join it from this link.