Selecting Multilingual for a model evaluation

[aladeen57]

Hello, thank you for the great work with mteb.

I'm currently trying to evaluate a sentence embedding model (multilingual-e5-large I have further finetuned) for the task retrival and I want to have a multilingual evaluation.
I'm sorry but I have issues finding how to specify multilingual in languages arg when selecting tasks :

tasks = mteb.get_tasks(task_types=["Retrieval"])

My goal is to compare my model to original model and all the others on the leadboard for retrival in multilingual.

Thanks in advance for your help

[Samoed]

You can select tasks from MTEB(Multilingual, v1)

import mteb

benchmark = mteb.get_benchmark("MTEB(Multilingual, v1)")
tasks = [task for task in benchmark.tasks if task.metadata.type == "Retrieval"]

[aladeen57]

Thanks a lot

[KennethEnevoldsen]

Another legit way of doing it is also:

import mteb

benchmark = mteb.get_benchmark("MTEB(Multilingual, v1)")
task_names = [task.metadata.name for task in benchmark.tasks]

tasks = mteb.get_tasks(tasks=task_names, task_types=["Retrieval")

Both methods are equivalent.