[Bug]: anomaly map normalization in efficient ad looks wrong

[jpcbertoldo]

Describe the bug

Poiting out a few things:

• A) it looks like the val split in the current yaml config of efficientad is cheating : )
• B) in its lightning module, the validation phase does the whole job in the start then behaves like a test phase
• C) the set of images used for the penalty in $L_{\text{ST}}$ is a small sample of imagenet

A) normalization params should depend on a split from the train set

The default YAML config is using a random split from the test set

https://github.com/openvinotoolkit/anomalib/blob/c97762c8781aacfc31d100ba862aa730a3262f7a/src/anomalib/models/efficient_ad/config.yaml#L18

which is usual for models that only do evaluation on the val set, but in efficientad

(from http://arxiv.org/abs/2303.14535, Section 3.4, page 6, left column)

☝️ the validation set is like a 2nd-phase (or 3rd counting the distillation) of the training, where $q^{\text{ST}}_a, q^{\text{ST}}_b, q^{\text{AE}}_a, q^{\text{AE}}_b$ are fitted.

It should be possible to split the train set and use it there.

---

B) validation start/step design is weird

I think lightning's design is made essentially to avoid loops on the batches.
By implementing validation_step(batch) you define what the model should do with that data -- which, for EfficientAD, is to find the $q$ parameters. In the current design, there is a for loop in the start

https://github.com/openvinotoolkit/anomalib/blob/c97762c8781aacfc31d100ba862aa730a3262f7a/src/anomalib/models/efficient_ad/lightning_model.py#L245-L252

function map_norm_quantiles()

https://github.com/openvinotoolkit/anomalib/blob/c97762c8781aacfc31d100ba862aa730a3262f7a/src/anomalib/models/efficient_ad/lightning_model.py#L154-L179

which is re-doing this loop-over-batches structure, while validation_step() is just predicting (which would happen in the test_step() or predict_step()):

https://github.com/openvinotoolkit/anomalib/blob/c97762c8781aacfc31d100ba862aa730a3262f7a/src/anomalib/models/efficient_ad/lightning_model.py#L253-L266

Side note: is it a good idea to have tqdm inside the model?

---

C) small imagenet

The data used in the penalty term of the student's loss function should be the same as the one from the teacher distillation (imagenet in the paper).

The code is currently downloading a reduced version with 10 classes.

The user should be able to use a custom folder (with imagenet already setup).

Code of Conduct

• I agree to follow this project's Code of Conduct

[alexriedel1]

Hey!
Thanks for adressing these points, let me clear some of them :)

A) it looks like the val split in the current yaml config of efficientad is cheating : )

Yes you're right about that. Fixing this isn't too easy in anomalib I guess, so the question is how this is affecting the model performance and if it is a real problem. Of course you shouldn't show the model anything of the validation set during training, but for this kind of normalization the model might not learn to much information from the validation data.

B) in its lightning module, the validation phase does the whole job in the start then behaves like a test phase

I don't get this completely. Why shouldn't it be designed this way? Using the tqdm in the model however seems fine to me, as long as its not in the forward function of the torch model (that is used for inference in the end)

C) the set of images used for the penalty in is a small sample of imagenet

Thats right, however the original paper also only uses a subset of 60k imagenet images..
again the question is, how does this affect the models performance.

[jpcbertoldo]

A) it looks like the val split in the current yaml config of efficientad is cheating : )

Yes you're right about that. Fixing this isn't too easy in anomalib I guess, so the question is how this is affecting the model performance and if it is a real problem. Of course you shouldn't show the model anything of the validation set during training, but for this kind of normalization the model might not learn to much information from the validation data.

I corrected that on my fork (will open the PR soon).
I am running a benchmark and the results are very disappointing (there's an visa class with AUPRO 0 🙃 ) so I'm wondering if this is coming from there -- I also had other two minor changes, so will roll them back before o check.

B) in its lightning module, the validation phase does the whole job in the start then behaves like a test phase

I don't get this completely. Why shouldn't it be designed this way? Using the tqdm in the model however seems fine to me, as long as its not in the forward function of the torch model (that is used for inference in the end)

I think lightning's design is made essentially to avoid loops on the batches.

By implementing validation_step(batch) you define what the model should do with that data -- which, for EfficientAD, is to find the $q$ parameters.

In the current design, there is a for loop in the start

https://github.com/openvinotoolkit/anomalib/blob/c97762c8781aacfc31d100ba862aa730a3262f7a/src/anomalib/models/efficient_ad/lightning_model.py#L168

which is re-doing this structure, while validation_step is really just predicting (which would happen in the test_step() or predict_step().

C) the set of images used for the penalty in is a small sample of imagenet

Thats right, however the original paper also only uses a subset of 60k imagenet images.. again the question is, how does this affect the models performance.

The 60k you refer to is from here, right?

(appendix A.2, algorithm 3, page 16)

Well, using the imagenet1k (1k classes, 1.2M images), would be quite more diverse then this subset of only 10 classes, but, yes, effect to be tested.

The paper's ablation suggests a somewhat important effect

This is one of the things I changed and will be rolling back to see the effect.

[jpcbertoldo]

I adressed A and C in #1376

[sjoerdvandenbos-prodrive]

#1376 looks like a nice PR, but it has been stuck in review without anything happening for more than a year now. I'm not a contributor in this repo, but is there anything I can do to help?