How evaluate only the Document Reader?

[leomaurodesenv]

I want to create a pipeline to evaluate only the Reader. I know we can create a ExtractiveQAPipeline and evaluate the the Reader isolated from the Retriever, using pipeline.eval(add_isolated_node_eval=True).

However, I want to create a pipeline with the Reader only and evaluate using the pipeline.eval() because I don't want to instantiate Document Retriever. Is it possible?

I'm trying to do something like

from haystack import Pipeline
from haystack.utils import print_answers

pipe = Pipeline()
pipe.add_node(component=reader, name='Reader', inputs=['Query'])

# Testing
question = "When was the Super Bowl anniversary?"
prediction = pipe.run(query=question, documents=docs[0:1], params={"Reader": {"top_k": 1}})
print_answers(prediction) # it is working fine

# Evaluation
eval_result = pipe.eval(labels=eval_labels, params={"Reader": {"top_k": 1}})

In evaluation, I got the error TypeError: run() missing 1 required positional argument: 'documents'. However, the eval_labels contains the Document and the rest. See one example

<MultiLabel: {'labels': [{'id': '84de18e7-7cc8-4f35-a7fa-24ee698ffd98', 'query': 'Which NFL team represented the AFC at Super Bowl 50?', 'document': {'content': 'Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super Bowl title. The game was played on February 7, 2016, at Levi\'s Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th Super Bowl, the league emphasized the "golden anniversary" with various gold-themed initiatives, as well as temporarily suspending the tradition of naming each Super Bowl game with Roman numerals (under which the game would have been known as "Super Bowl L"), so that the logo could prominently feature the Arabic numerals 50.', 'content_type': 'text', 'id': '327347753074272752500655137676769839737', 'meta': {}, 'score': None, 'embedding': None}, 'is_correct_answer': True, 'is_correct_document': True, 'origin': 'gold-label', 'answer': {'answer': 'Denver Broncos', 'type': 'extractive', 'score': None, 'context': None, 'offsets_in_document': None, 'offsets_in_context': None, 'document_id': None, 'meta': {}}, 'no_answer': False, 'pipeline_id': None, 'created_at': '2023-01-29 22:21:29', 'updated_at': None, 'meta': {}, 'filters': None}], 'query': 'Which NFL team represented the AFC at Super Bowl 50?', 'answers': ['Denver Broncos'], 'no_answer': False, 'document_ids': ['327347753074272752500655137676769839737'], 'contexts': ['Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super Bowl title. The game was played on February 7, 2016, at Levi\'s Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th Super Bowl, the league emphasized the "golden anniversary" with various gold-themed initiatives, as well as temporarily suspending the tradition of naming each Super Bowl game with Roman numerals (under which the game would have been known as "Super Bowl L"), so that the logo could prominently feature the Arabic numerals 50.'], 'offsets_in_contexts': [], 'offsets_in_documents': []}>

[bogdankostic]

Hi @leomaurodesenv, you need to pass the Documents explicitly to the Reader when it isn't preceded by a Retriever node. You can do it for example like this:

eval_docs = [[label.document for label in multi_label.labels] for multi_label in eval_labels]
eval_result = pipe.eval(labels=eval_labels, documents=eval_docs)

[leomaurodesenv]

It worked, thanks a lot