PerceptualEvaluationSpeechQuality - NoUtterancesError: b'No utterances detected'

[maxstrobel]

🐛 Bug

PerceptualEvaluationSpeechQuality raises for some samples `NoUtterancesError: b'No utterances detected'.

To Reproduce

Sample code and steps to reproduce the behavior with expected result...

Code sample

import torch
from torchmetrics.audio.pesq import PerceptualEvaluationSpeechQuality

target = torch.load("target.pt")
print(f"{target.shape=} {target.dtype=} {target.min()=} {target.max()=} {target.mean()=} {target.std()=}")
pred = torch.load("pred.pt")
print(f"{pred.shape=} {pred.dtype=} {pred.min()=} {pred.max()=} {pred.mean()=} {pred.std()=}")


pesq = PerceptualEvaluationSpeechQuality(fs=8000, mode="nb")
pesq.update(pred, target)

Environment

• TorchMetrics version (if build from source, add commit SHA): 1.7.0
• Python & PyTorch Version (e.g., 1.0): 3.11.10 & 2.6.0+cpu (also CUDA version)
• Any other relevant information such as OS (e.g., Linux): Windows & Linux

Additional context

Attached a data sample to reproduce the issue.

sample_data.zip

[VijayVignesh1]

Hi @maxstrobel
After investigating the issue, I found that the main cause is a range mismatch between pred and target. Here are the statistics:

target.min()=tensor(-0.4522) target.max()=tensor(0.3976) target.mean()=tensor(-0.0001) target.std()=tensor(0.0350)

pred.min()=tensor(-156.1769) pred.max()=tensor(148.0738) pred.mean()=tensor(-0.0428) pred.std()=tensor(11.1761)

The PESQ function normalizes both signals using the maximum absolute value between the reference and degraded signals, as shown below (reference: https://github.com/ludlows/PESQ/blob/master/pesq/_pesq.py):

max_val = max(np.max(np.abs(ref / 1.0)), np.max(np.abs(deg / 1.0))) 
ref = ref / max_val 
deg = deg / max_val

Since the prediction (pred) has a much larger amplitude range than the target, this normalization scales the target values very close to zero. As a result, PESQ’s internal utterance detection (VAD) fails to detect any valid speech activity, triggering the “No utterances detected” error in certain cases.

Hope that helps!

[maxstrobel]

@VijayVignesh1 Thanks for the explanation - this makes indeed sense.

I checked the paper PERCEPTUAL EVALUATION OF SPEECH QUALITY (PESQ) – A NEW METHOD FOR SPEECH QUALITY ASSESSMENT OF TELEPHONE NETWORKS AND CODECS (Rix et al.) where they mention just

The model begins by level aligning both signals to a standard listening level.

Is the normalization you mentioned a standard way of doing this?
An alternative approach would be to align both signals to a specific loudness level e.g. -23 LUFS, which would avoid this behaviour.

What do you think?