Add Support for Closed-Source ASR Models

[ahmed-haytham0]

Hello,

We have developed a closed-source ASR model which performs exceptionally well on dialects and Modern Standard Arabic (MSA). We believe that once added, it will be the first on the list.

Using your codebase, we conducted an evaluation on the SADA dataset and achieved the following results:

• Word Error Rate (WER): 0.37
• Character Error Rate (CER): 0.22

Detailed metrics (SADA DATASET):

WER: 0.37371696623473133
CER: 0.22221457671436673
(WER, CER): (0.37371696623473133, 0.22221457671436673)

Request:

We would like to request the ability to add closed-source ASR models to the leaderboard. Please let us know if there are any guidelines or additional steps needed to support closed-source entries. Shall we continue the evaluation?

Thank you

[BenoitWang]

Hello @ahmed-haytham0, thank you for your interest. However, similar as in #2, closed-sourced models cannot assure that the pre-training or fine-tuning datasets are not in-domain of our test sets. We are sorry that for fairness we are not planning to include such models, but still I suggest you to run the evaluation on the other datasets to see the overall performance of your model.

[ahmed-haytham0]

Hello @BenoitWang ,
Thank you for your response. I’d like to clarify that we did not use the test datasets in our model’s pre-training or fine-tuning.
We are happy to continue the evaluation if it would be beneficial for adding the model to the leaderboard.
Anyway, thank you!

[BenoitWang]

Hi @ahmed-haytham0, the in-domain issue is not about the test set but about the training set. For example if SADA training set is used in model fine-tuning, surely the model will have better results in SADA test set. Because the training set and test set are in the same domain (distribution). This is a typical concern for leaderboards, so we are sorry that for the leaderboard it may not be appropriate. Thank you again for your interest!