Error in calculate_metric_plain method?

[LaurenzV]

Hello,

I'm trying to use the calculate_metric_plain method to calculate the necessary metrics for our models. However, upon running the method, I get the following error:

Traceback (most recent call last):
  File "C:/Users/laure/Desktop/air-2022-group-02/src/re_ranking.py", line 217, in <module>
    _evaluate(config["test_data"], config["test_qrels"])
  File "C:/Users/laure/Desktop/air-2022-group-02/src/re_ranking.py", line 209, in _evaluate
    metrics = calculate_metrics_plain(results, qrels)
  File "C:\Users\laure\Desktop\air-2022-group-02\src\core_metrics.py", line 32, in calculate_metrics_plain
    rr_per_candidate_depth = np.zeros(len(global_metric_config["MRR+Recall@"]), ranked_queries)
TypeError: Cannot interpret '2000' as a data type

Upon taking a look at the code, it makes sense to me why:

ranked_queries = len(ranking)
ap_per_candidate_depth = np.zeros(ranked_queries)
rr_per_candidate_depth = np.zeros(len(global_metric_config["MRR+Recall@"]), ranked_queries)
rank_per_candidate_depth = np.zeros(len(global_metric_config["MRR+Recall@"]), ranked_queries)
recall_per_candidate_depth = np.zeros(len(global_metric_config["MRR+Recall@"]), ranked_queries)
ndcg_per_candidate_depth = np.zeros(len(global_metric_config["nDCG@"]), ranked_queries)

The ranked_queries is passed as a second attribute to np.zeros(), meaning that it interprets it as the dtype of the array, even though it should still be part of the shape. Changing the code the following way fixes the issue for me:

ranked_queries = len(ranking)
ap_per_candidate_depth = np.zeros(ranked_queries)
rr_per_candidate_depth = np.zeros((len(global_metric_config["MRR+Recall@"]), ranked_queries))
rank_per_candidate_depth = np.zeros((len(global_metric_config["MRR+Recall@"]), ranked_queries))
recall_per_candidate_depth = np.zeros((len(global_metric_config["MRR+Recall@"]), ranked_queries))
ndcg_per_candidate_depth = np.zeros((len(global_metric_config["nDCG@"]), ranked_queries))

However, I also checked the assignment template from last year and it had the same piece of code, so I think such an issue wouldn't have gone unnoticed. Is there maybe something else I'm doing wrong? Do I need to use a lower version of numpy (although there was none in the requirements.txt)?

Kind regards,
Laurenz

[sophiaalthammer]

Dear Laurenz,

thanks for bringing up this issue, this is definitely a mistake in the published code and your proposed solution is the right one!

Best,
Sophia

[Darius888]

Hello,

Which version have you used ?

Best,

Darius

[LaurenzV]

What do you mean which version have I used?

[Darius888]

Accepted answer is that "your proposed solution is the right one!" and from your text I understood that we need to lower the numpy version to avoid "Cannot interpret '2000' as a data type" , is that correct ?

Best,

Darius

[LaurenzV]

No, you just need to add the brackets to the method calls, just like in the second code snippet.

[Darius888]

Ahhhh, yes, that's what happens, when you don't read everything. :D Thanks a lot !

Best,

Darius