Ex. 1: "xy" === "xy definition"?

[Sejmou]

Hi,
sorry if this question comes a bit late. I am still a bit unsure whether I am doing the annotation task correctly.
Up until now, I was often not sure whether I should mark a query result as "perfect", or only "partial" or "topic".

The main reason for this is that some queries only include keywords, but no additional contextual information. Instead of "xy definition" or "cause of xy", only "xy " is given. Should all those queries be interpreted as "xy definition"?

For example, should I mark the result for "smoking cessation" only as perfect as long as a definition of the term is given? Could you maybe also give examples for "partial" or "topic" results for this query?

The example for an exact match given on the Fira website is a bit more specific, as the query explicitly asks for a definition of a given term, so I just wanted to check back. I really don't want to produce useless annotations.

UPDATE: I just found the following passage in the notes for exercise 1 on TUWEL:

We have quite a few shorter keyword based queries in this dataset -> here it might be difficult to select a grade to annotate, when in doubt - if the passage partially talks about the subject - choose partially relevant. If the keywords are only mentioned, but not center of the topic talked about choose topically related.

So, in the case of a rather "unspecific" query only consisting of the name of some medical condition xy, "topic" matches are "impossible", and we should always just go with "partial" instead, right?

Best,
Samo

[sebastian-hofstaetter]

Hi,
I would say that you don't have to interpret the query too much - if xy is talked about in the document (even if it's not a definition) then I would select it as partial. Only if xy (or only x) appears in the text somewhat randomly as a side note, then it's topical. If it doesn't appear at all (or only 1 of multiple terms, that have clearly nothing to do with the full keyword query) then it is wrong.

It is ok if a query never gets assigned a perfect label -> that is why for example the ndcg metric normalizes based on the best possible result (the best possible result may only consist of partial and topic documents).

Best,
Sebastian