Ex2 Part 1 - Different ids for same text in query/doc #90
-
|
Hi, We saw that there are different ids for the same query texts in some cases (e.g. "financial times" ). Same happens for some pairs of doc ids and doc texts.
Could we aggregate all the ids for one such text sample? This would increase the number of judgements we have per pair query_id <-> doc_id. Best, |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 1 reply
-
|
Hi, thats a good and interesting finding! But I would not recommend aggregating the query ids, so that the test collections to the other teams remain comparable! Best, |
Beta Was this translation helpful? Give feedback.
-
|
Our team also aggregated the same queries/documents, but we've already done the evaluation of the three models... Is this a big issue and will it lead to point deductions if we leave it like that? |
Beta Was this translation helpful? Give feedback.
Hi,
thats a good and interesting finding! But I would not recommend aggregating the query ids, so that the test collections to the other teams remain comparable!
Best,
Sophia