ArTST Model Request

[djanibekov]

Please find links to the papers of the requested model

Dialectal ArTST: https://arxiv.org/abs/2411.05872
MSA ArTST: https://aclanthology.org/2023.arabicnlp-1.5/

GitHub link: https://github.com/mbzuai-nlp/ArTST?tab=readme-ov-file

Please let me know if you need help running evaluation as we already have some datasets.

[BenoitWang]

Hello @djanibekov, sorry for the late reply and thank you for your interest. Just read your papers really appreciate your work! However if I made it right, there's an issue when I tried to fetch the weights of ArTST v2 ASR (QASR) through HF, while that of ArTST v2 ASR (Dialect) is not open-sourced yet.

Another thing I noted in your paper is that the model has already been pre-trained on mgb-2 and common-voice, then fine-tuned on mgb-2, SADA and MASC. The test sets of these datasets are used in our benchmark so there's an in-domain issue and the results can be biased. In order to keep the fairness of the leaderboard we don't think we'll include the model but I can still show you the numbers here once the HF issue is solved.

[djanibekov]

@BenoitWang, you can try inferencing our ArTST v1 (MSA) model on SADA and MASC but do not consider the results of MGB2. Currently, we have a problem with HF-based models. We will let you know when we will fix the issue. Thanks!

[djanibekov]

btw, @BenoitWang you can still use our checkpoints that been not fine-tuned on SADA, MASC, in particular, our v2 QASR or MGB2 checkpoints for your leaderboard.

ArTST v1 fine-tuned on MGB2: https://huggingface.co/MBZUAI/artst_asr
ArTST v2 fine-tuned on MGB2 only: https://huggingface.co/MBZUAI/artst_asr_v2
ArTST v3 fine-tuned on MGB2 only: https://huggingface.co/MBZUAI/artst_asr_v3

We still figuring out huggingface integration for model fine-tuned with v2 QASR.

[BenoitWang]

Hi @djanibekov, just tried the 3 HF links you sent, but they all output empty strings for whatever audio inputs, am I missing something please? As for the leaderboard yes v2 QASR model will be the most appropriate.

[djanibekov]

Are you audios that you try in 16kHz?. Actually you can use our inference code that we use for our experiments.

import os
import json
import time

import torch
import warnings
warnings.filterwarnings("ignore")


import soundfile as sf
from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor, pipeline

from loguru import logger
from tqdm import tqdm

import re
import sys
import unicodedata

def remove_punctuation(word):
    punctuations = ''.join([chr(i) for i in list(i for i in range(sys.maxunicode) if unicodedata.category(chr(i)).startswith('P'))])
    return word.translate(str.maketrans('', '', re.sub('[@% ]','', punctuations))).lower()


def run_artst(model_id, data_manifest, data_folder, output_manifest):
    """"
    Arguments
    ---------
    model_id: str
        HuggingFace whisper model name
    data_manifest: str
        Path of a data manifest under datasets/
    data_folder: str
        The path to the test set
    output_manifest: str
        The output manifest path

    Output
    ---------
    Create an output manifest containing ground truths and predictions
    
    Example:

    # MBZUAI/artst_asr_v2
    # MBZUAI/artst_asr_v2_qasr

    processor = AutoProcessor.from_pretrained("MBZUAI/artst_asr_v2")
    model = AutoModelForSpeechSeq2Seq.from_pretrained("MBZUAI/artst_asr_v2").to(device)
    
    """
    device = 'cuda' if torch.cuda.is_available() else 'cpu'
    torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32

    processor = AutoProcessor.from_pretrained(model_id)
    model = AutoModelForSpeechSeq2Seq.from_pretrained(model_id).to(device)

    pipe = pipeline(
        "automatic-speech-recognition",
        model=model,
        tokenizer=processor.tokenizer,
        feature_extractor=processor.feature_extractor,
        torch_dtype=torch_dtype,
        device=device,
    )

    with open(data_manifest, "r") as f:
        with open(output_manifest, 'w') as fout:
            all_inference_time = 0
            all_audio_duration = 0
            all_inference_memory = []
            count = 0
            counter = 1
            for i, line in enumerate(tqdm(f)):
                item = json.loads(line)
                in_path = item["audio_filepath"].format(data_folder=data_folder)
                duration = item["duration"]
                reference = item['text']

                torch.cuda.reset_max_memory_allocated(torch.device("cuda"))
                initial_memory = torch.cuda.max_memory_allocated(torch.device("cuda"))/(1024 ** 3)

                try:
                    wav, samplerate = sf.read(in_path)
                except:
                    logger.error(in_path)
                    continue
                assert samplerate == 16000

                start_time = time.time()
                
                result = pipe(wav, generate_kwargs={'num_beams': 5, 'early_stopping': True})
                transcription = remove_punctuation(result["text"])
                end_time = time.time()
                
                inference_time = end_time - start_time
                if count > 4:
                    all_inference_time += inference_time
                    all_audio_duration += duration
                peak_memory = torch.cuda.max_memory_allocated(torch.device("cuda"))/(1024 ** 3)
                all_inference_memory.append(peak_memory-initial_memory)        
                count += 1

                metadata = {
                    "audio_filepath": in_path,
                    "text": item["text"],
                    "pred_text": transcription,
                }
                json.dump(metadata, fout, ensure_ascii=False)
                fout.write('\n')
        
        print("average rtf : ", all_inference_time/all_audio_duration)
        print("model memory : ", initial_memory)
        print("average inference-only memory : ", sum(all_inference_memory)/len(all_inference_memory))

[BenoitWang]

Hi @djanibekov yes I used the same code. It was the transformers version, after updating everything was fine. I ran the evaluation of MBZUAI/artst_asr_v2 and I got an average wer over 100%, even though the wer on mgb2 is only 13.49%.

The fact is that it hallucinates quite a lot on SADA, MASC-test-noisy and Casablanca, below are some samples of SADA.

You can also infer your model on these datasets to verify.

[djanibekov]

@BenoitWang we will test ourself on datasets that are in this repo and prepare python runnable files

[djanibekov]

@BenoitWang, thank you very much for pointing out the issue with SADA. We found a mistake in our paper with the results on SADA. We will update arxiv version ASAP.

P.S. I have updated code above to be compatible with your repository. We will run experiments on test sets and give performance results ASAP.

[BenoitWang]

Hey @djanibekov, glad that I helped. Looking forward to your improved version. The model trained on QASR will be a good addition into the leaderboard.