Model comparison tool?

[jasontitus]

Has anyone written or released a tool for comparing the WER of various models against a given dataset? It seems like as more and more fine tuning happens, folks will want to know which model is best for their language or use case. I have been tinkering with Common Voice sets and doing short run fine tuning against them but that is pretty high overhead and not automated at all.

Ideally I could just point to a set of local directories or hugginface repositories and set the evaluation dataset and compute metric.

Just checking before I hack together something of my own

[Alumniminium]

here