Evaluation tab - BirdNET GUI

[Bry0ny]

Hi all, apologies if this has been answered elsewhere.

I'm hoping someone could help run me through how to use the "Evaluation" tab in the BIRDNET GUI?
I've built a series of custom classifiers that I wish to test, and whilst I have developed some R script to generate various statistics of model performance (precision, recall, F1 Scores, AUROC, etc.) I'd like to compare my results with the GUI outputs.

Specifically, I'm a little confused about what file formats need to be loaded under "select annotation directory" and "select prediction directory" and what the annotation columns and prediction columns should be titled.
I'm getting results but they aren't correct - Looking at the data table and confusion matrix, it seems my positive class is uploading fine, but it doesn't seem to be recognising my negative class, despite trialling multiple scenarios

Any help would be greatly appreciated!!

[max-mauermann]

https://birdnet-team.github.io/BirdNET-Analyzer/best-practices/evaluation-tool.html

We updated our documentation to include a guide to this tool written by @reheinrich who also implemented this feature.

From what i can tell the columns containing the name of the predicted/annotated species can be selected in the GUI and do not require a specific name. (but they should be equal across all annotation files).

[Bry0ny]

Hi Max, huge thanks for this! I'll see how I go with those instructions

[Bry0ny]

I managed to get the annotation and prediction directories to combine. They now create a data table with times aligned, the confidence score for a single species (or zero when no prediction is made), and an annotation column (0 or 1).
Interestingly, it also includes the same for the negative class: an annotations column (0 or 1) that is the reverse of the positive class (so looks good) and then a confidence score column that contains only zeros.

The output is giving me the correct recall and precision results now, so that's great!

However, the confusion matrix is still incorrect. It lists TN=1, FP=0, which is incorrect. The FN and TP are proportionally correct. It seems the issue likely lies in how it's treating/how I'm inputting the negative class? Likely, I am not combining the two files in the correct way.

Apologies for the trouble, but @reheinrich if you have any suggestions or perhaps an example you could share, it would be much appreciated! It seems like a great tool for assessing my different custom classifiers, so I'd love to know how to use it correctly!

Heres a small screenshot as an example of the combined data table:

[reheinrich]

Hi @Bry0ny, thanks for the example, and great that recall and precision now match!

The Evaluation tab shows a row-normalized confusion matrix (each row sums to 1).
Rows are the true class (0 = negative, 1 = positive); columns are the predicted class (0/1).

Based on your EP annotations, the confusion matrix in absolute counts for your snippet (EP as positive, threshold = 0.1) is:

[[TN=1, FP=0],
 [FN=13, TP=2]]

When the confusion matrix is row-normalized (what the Evaluation tab displays), it becomes:

[[1.00, 0.00],
 [0.87, 0.13]]

So “TN=1, FP=0” in the Evaluation tab means 100% of true negatives were predicted negative (i.e., score below the threshold).

I hope this helps clarify things.

Best,
René

[Bry0ny]

Hi René,

Many thanks for your help. I can see the problem with my interpretation now.
Essentially, the ratio of my TN(1085) to FP(3) was so high that the row-normalised matrix is rounding TN=0.997 and FP=0.002 to TN=1, FP=0. I assumed the matrix wasn't accounting for them, when it was.

Thank you very much for helping me here - very much appreciated!