Overlap vs. Context Mode

[Mattk70]

The classifiers (models) used by Chirpity segment audio into 3 second chunks and analyse a spectrogram (or 2 in the case of BirdNET) generated from that audio segment. By default each segment is in sequence with no overlap. This is generally fine, however, because the model 'sees' only 3 seconds of audio, there is no context for the classification and sometimes a call which lies at the edge of one of those windows is cut short.

Although this case is factored into model training, that can still have one of two undesirable effects:

1. The species' call fragment may not be recognised in either segment of audio
2. The fragment may resemble the call of another species, so a false positive ID is given.

Two approaches can be used to mitigate these problems. They both involve overlapping the segments:

The simple approach is to analyse these overlapping segments and report the results for each segment. This addresses issue 1, but increases the occurrence of problem 2.

In Chirpity, a different approach is available for the Nocmig models. This is "Context mode" :

The model generates predictions for each 3 second audio segment, and compares these to the predictions of the surrounding audio - offset by 1.5 seconds before and after the current segment.
The results are compared and any result that does not meet the chosen confidence threshold in 2 of the 3 segments is discarded. This approach mitigates both issues 1 and 2, although a lower confidence threshold is usually required for best results.

N.B. Because of the additional processing, you can expect Context Mode to take about 50% longer to complete an analysis.

[ceperman]

(or 2 in the case of BirdNET)
Is it not 3 sec for BirdNET also? I thought this was fixed for BN, and the segments shown in the spectrogram are 3 secs long.

[Mattk70]

Two spectrograms, not 2 second spectrograms, BirdNETs are also 3 seconds long.

[ceperman]

Two spectrograms, not 2 second spectrograms, BirdNETs are also 3 seconds long.

Ah, misunderstanding.

Would it be possible to provide a context mode for BirdNET (assuming resources were available of course)? I do suffer from false positives for the reason you explain. I did try using the overlap feature when I was using BirdNET directly before coming to Chirpity, but not enough to come to any conclusion about its use.

[Mattk70]

Everything is possible, but in this case it’s unlikely. BirdNET takes audio input , whereas the Nocmig models accept spectrogram input. It’s easy and fast to shift the window on the spectrogram, you just slice it in different positions. You can do this with audio too, but giving that audio to birdNET would result in inference taking 3x longer, not the 50% penalty seen for the nocmig models.

One could rearchitect BidNET’s internals to do something similar, but I don’t want to go there.

[ceperman]

Unlikely because you don't think it's worth it? A 50% overlap (1.5 secs) would double the number of analysed intervals which presumably would double the BirdNET analysis time, so with interpretation time I can see how that heads towards a 3x overhead. But with processor speed increasing all the time, perhaps this is not such a big deal. Some may think it worth it. Is it possible to quantify the improvement of using context mode?