Nvidia nemo automatic speech recognition

[coolvs4]

Good afternoon, can I, when calling the transcribe method, pass not a list of paths to audio files, but a raw audio file in the form of an array? Reading the documentation, I found out that there is no such built-in function. Can I change the source code of nemo libraries? Thanks for your attention.

asr_model.transcribe(paths2audio_files=['test.wav'])

[titu1994]

We currently don't support passing audio buffer and only support files. In the future we may support audio buffers

[titu1994]

As to changing source code, sure ! You can subclass and override your transcribe() method quite easily by copy pasting the original code and modifying what you need.

[bhargaviajaypatel]

Good afternoon, can I, when calling the transcribe method, pass not a list of paths to audio files, but a raw audio file in the form of an array? Reading the documentation, I found out that there is no such built-in function. Can I change the source code of nemo libraries? Thanks for your attention.

asr_model.transcribe(paths2audio_files=['test.wav'])

Can you provide the code ? If you have made the changes.

Thank you.

[coolvs4]

Hello.
I implemented it this way
In the variable audio_buf I get a byte representation of the audio file in the format .wav
Next, I perform the following transformation:

from subprocess import Popen, PIPE
import io
import torchaudio
import torch

proc = Popen(cmd, stdout=PIPE, stderr=PIPE, stdin=PIPE)
result, stderr = proc.communicate(input=audio_buf)
for_tensor = io.BytesIO(result)
waveform, _ = torchaudio.load(for_tensor)
len_waveform = torch.tensor([len(waveform[0])])

Next, I call the redefined method for the model transcribe
models.transcribe(tensor_wav=[[waveform, len_waveform]], logprobs=True)

The source code of the method, an example for EncDecCTCModel:

@torch.no_grad()
    def transcribe(
        self,
        paths2audio_files: List[str],
        batch_size: int = 4,
        logprobs: bool = False,
        return_hypotheses: bool = False,
        num_workers: int = 0,
        channel_selector: Optional[ChannelSelectorType] = None,
    ) -> List[str]:
        """
        Uses greedy decoding to transcribe audio files. Use this method for debugging and prototyping.

        Args:
            paths2audio_files: (a list) of paths to audio files. \
                Recommended length per file is between 5 and 25 seconds. \
                But it is possible to pass a few hours long file if enough GPU memory is available.
            batch_size: (int) batch size to use during inference.
                Bigger will result in better throughput performance but would use more memory.
            logprobs: (bool) pass True to get log probabilities instead of transcripts.
            return_hypotheses: (bool) Either return hypotheses or text
                With hypotheses can do some postprocessing like getting timestamp or rescoring
            num_workers: (int) number of workers for DataLoader
            channel_selector (int | Iterable[int] | str): select a single channel or a subset of channels from multi-channel audio. If set to `'average'`, it performs averaging across channels. Disabled if set to `None`. Defaults to `None`.

        Returns:
            A list of transcriptions (or raw log probabilities if logprobs is True) in the same order as paths2audio_files
        """
        if paths2audio_files is None or len(paths2audio_files) == 0:
            return {}

        if return_hypotheses and logprobs:
            raise ValueError(
                "Either `return_hypotheses` or `logprobs` can be True at any given time."
                "Returned hypotheses will contain the logprobs."
            )

        if num_workers is None:
            num_workers = min(batch_size, os.cpu_count() - 1)

        # We will store transcriptions here
        hypotheses = []
        all_hypotheses = []

        # Model's mode and device
        mode = self.training
        device = next(self.parameters()).device
        dither_value = self.preprocessor.featurizer.dither
        pad_to_value = self.preprocessor.featurizer.pad_to

        try:
            self.preprocessor.featurizer.dither = 0.0
            self.preprocessor.featurizer.pad_to = 0
            # Switch model to evaluation mode
            self.eval()
            # Freeze the encoder and decoder modules
            self.encoder.freeze()
            self.decoder.freeze()
            logging_level = logging.get_verbosity()
            logging.set_verbosity(logging.WARNING)
            # Work in tmp directory - will store manifest file there
            with tempfile.TemporaryDirectory() as tmpdir:
                with open(os.path.join(tmpdir, 'manifest.json'), 'w', encoding='utf-8') as fp:
                    for audio_file in paths2audio_files:
                        entry = {'audio_filepath': audio_file, 'duration': 100000, 'text': ''}
                        fp.write(json.dumps(entry) + '\n')

                config = {
                    'paths2audio_files': paths2audio_files,
                    'batch_size': batch_size,
                    'temp_dir': tmpdir,
                    'num_workers': num_workers,
                    'channel_selector': channel_selector,
                }

                temporary_datalayer = self._setup_transcribe_dataloader(config)
                for test_batch in tqdm(temporary_datalayer, desc="Transcribing"):
                    logits, logits_len, greedy_predictions = self.forward(
                        input_signal=test_batch[0].to(device), input_signal_length=test_batch[1].to(device)
                    )
                    if logprobs:
                        # dump log probs per file
                        for idx in range(logits.shape[0]):
                            lg = logits[idx][: logits_len[idx]]
                            hypotheses.append(lg.cpu().numpy())
                    else:
                        current_hypotheses, all_hyp = self.decoding.ctc_decoder_predictions_tensor(
                            logits, decoder_lengths=logits_len, return_hypotheses=return_hypotheses,
                        )

                        if return_hypotheses:
                            # dump log probs per file
                            for idx in range(logits.shape[0]):
                                current_hypotheses[idx].y_sequence = logits[idx][: logits_len[idx]]
                                if current_hypotheses[idx].alignments is None:
                                    current_hypotheses[idx].alignments = current_hypotheses[idx].y_sequence

                        hypotheses += current_hypotheses

                        # Keep following for beam search integration
                        # if all_hyp is not None:
                        #     all_hypotheses += all_hyp
                        # else:
                        #     all_hypotheses += current_hypotheses

                    del greedy_predictions
                    del logits
                    del test_batch
        finally:
            # set mode back to its original value
            self.train(mode=mode)
            self.preprocessor.featurizer.dither = dither_value
            self.preprocessor.featurizer.pad_to = pad_to_value
            if mode is True:
                self.encoder.unfreeze()
                self.decoder.unfreeze()
            logging.set_verbosity(logging_level)

        return hypotheses

Modified code:

@torch.no_grad()
    def transcribe(
            self,
            tensor_wav: List,
            # paths2audio_files: List[str],
            batch_size: int = 4,
            logprobs: bool = False,
            return_hypotheses: bool = False,
            num_workers: int = 0,
    ) -> List[str]:
        """
        Uses greedy decoding to transcribe audio files. Use this method for debugging and prototyping.

        Args:
            paths2audio_files: (a list) of paths to audio files. \
                Recommended length per file is between 5 and 25 seconds. \
                But it is possible to pass a few hours long file if enough GPU memory is available.
            batch_size: (int) batch size to use during inference.
                Bigger will result in better throughput performance but would use more memory.
            logprobs: (bool) pass True to get log probabilities instead of transcripts.
            return_hypotheses: (bool) Either return hypotheses or text
                With hypotheses can do some postprocessing like getting timestamp or rescoring
            num_workers: (int) number of workers for DataLoader

        Returns:
            A list of transcriptions (or raw log probabilities if logprobs is True) in the same order as paths2audio_files
        """
        # if paths2audio_files is None or len(paths2audio_files) == 0:
        #     return {}

        if return_hypotheses and logprobs:
            raise ValueError(
                "Either `return_hypotheses` or `logprobs` can be True at any given time."
                "Returned hypotheses will contain the logprobs."
            )

        if num_workers is None:
            num_workers = min(batch_size, os.cpu_count() - 1)

        # We will store transcriptions here
        hypotheses = []
        all_hypotheses = []

        # Model's mode and device
        mode = self.training
        device = next(self.parameters()).device
        dither_value = self.preprocessor.featurizer.dither
        pad_to_value = self.preprocessor.featurizer.pad_to

        try:
            self.preprocessor.featurizer.dither = 0.0
            self.preprocessor.featurizer.pad_to = 0
            # Switch model to evaluation mode
            self.eval()
            # Freeze the encoder and decoder modules
            self.encoder.freeze()
            self.decoder.freeze()
            logging_level = logging.get_verbosity()
            logging.set_verbosity(logging.WARNING)
            temporary_datalayer = tensor_wav
            for test_batch in tqdm(temporary_datalayer, desc="Transcribing"):
                logits, logits_len, greedy_predictions = self.forward(
                    input_signal=test_batch[0].to(device), input_signal_length=test_batch[1].to(device)
                )
                if logprobs:
                    # dump log probs per file
                    for idx in range(logits.shape[0]):
                        lg = logits[idx][: logits_len[idx]]
                        hypotheses.append(lg.cpu().numpy())
                else:
                    current_hypotheses, all_hyp = self.decoding.ctc_decoder_predictions_tensor(
                        logits, decoder_lengths=logits_len, return_hypotheses=return_hypotheses,
                    )

                    if return_hypotheses:
                        # dump log probs per file
                        for idx in range(logits.shape[0]):
                            current_hypotheses[idx].y_sequence = logits[idx][: logits_len[idx]]
                            if current_hypotheses[idx].alignments is None:
                                current_hypotheses[idx].alignments = current_hypotheses[idx].y_sequence

                    hypotheses += current_hypotheses

                    # Keep following for beam search integration
                    # if all_hyp is not None:
                    #     all_hypotheses += all_hyp
                    # else:
                    #     all_hypotheses += current_hypotheses

                del greedy_predictions
                del logits
                del test_batch
        finally:
            # set mode back to its original value
            self.train(mode=mode)
            self.preprocessor.featurizer.dither = dither_value
            self.preprocessor.featurizer.pad_to = pad_to_value
            if mode is True:
                self.encoder.unfreeze()
                self.decoder.unfreeze()
            logging.set_verbosity(logging_level)

        return hypotheses

It looks pretty crutchy, everything was decided in a hurry.

[coolvs4]

I almost forgot, this is what the cmd list for changing the audio format looks like
cmd = ['ffmpeg', '-i', 'pipe:0', '-map_metadata', '-1', '-fflags', '+bitexact','-ar', '16000', '-ac', '1','-f', 'wav', 'pipe:1']