Adjusted mean rank

[cthoyt]

@mberr's adjusted mean rank address some of the problems with the mean rank, including its size dependence. Reference: https://arxiv.org/abs/2002.06914

[benedekrozemberczki]

We will add it to the library! Thank you @cthoyt :)

[cthoyt]

Reference implementation in PyKEEN here: https://github.com/pykeen/pykeen/blob/139c3d1db77c9e443d03db5383a163297345adfd/src/pykeen/evaluation/rank_based_evaluator.py#L468-L478

[mberr]

I am one of the authors and happy to contribute an implementation. To this end, I have a question regarding the existing metrics, e.g.,

rexmex/rexmex/metrics/ranking.py

Lines 64 to 73 in 8e5d0ad

	def mean_rank(relevant_items: Sequence[X], recommendation: Sequence[X]) -> float:
	"""
	Calculate the arithmetic mean rank (MR) of items in a ranked list.
	Args:
	relevant_items: An N x 1 sequence of relevant items.
	predicted: An N x 1 sequence of ordered items.
	Returns:
	: The mean rank of the relevant items in a predicted.
	"""

If I understand the API correctly, you assume that there is a single recommendation, a sequence of items, and some of them are relevant (relevant_items). The mean in mean rank is then computed over the relevant items.

This seems to be a bit different than the evaluation protocol frequently used in (Knowledge Graph) Link Prediction, where we have multiple recommendations (e.g. for each head-relation pair), each with relevant items (the tail entities), and compute the mean over all of these.