✨ Multi-dataset General ood eval

[fira7s]

• Support for multi-dataset OOD eval in classification routine
• Each datamodule should define near and far OOD datasets to be used by default
• According to args defined in datamodule (OOD and shift), routine is automatically configured Not possible
• Add SOTA OOD eval techniques from OpenOOD
• fix ugly output + add id accuracy using the ood criteria Not needed for now
• fix imagent 1k eval shift
• set ood datasets names in output table when using custom datasets
• add ood datasets to huggingface
• Add imagenet 200
• hyperparam search for some ood postprocessors
• Fix Mnist perf
• Add ood tutorial
• fix docstrings
• fill coverage

[o-laurent]

@fira7s you can install the pre-commits with pre-commit install to avoid linting errors in the CI 🙂

[fira7s]

My bad i knew it would cause the error but i ignored it @o-laurent i will be also changing more stuff in the routine and datamodules so i may need your opinion after it's done i hope my changes won't break other parts of the library.

[o-laurent]

Don't worry @fira7s , it was just a piece of friendly advice! I'll have a look, thanks!

[o-laurent]

Thanks for your work @fira7s
We should put all the configuration files on HuggingFace 😉

[fira7s]

@o-laurent Yes indeed, i'm working on merging this into dev right now, i will put all config files on huggingface once i'm finished.

[o-laurent]

(NB: We'll need to squash the PR)

[codecov]

Codecov Report

Attention: Patch coverage is 39.02066% with 797 lines in your changes missing coverage. Please review.

Project coverage is 89.89%. Comparing base (61791aa) to head (59a0240).
Report is 9 commits behind head on dev.

✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
torch_uncertainty/ood/ood_criteria.py	36.19%	303 Missing and 2 partials ⚠️
...ertainty/datamodules/classification/imagenet200.py	21.71%	119 Missing ⚠️
torch_uncertainty/routines/classification.py	61.57%	61 Missing and 12 partials ⚠️
torch_uncertainty/ood/nets/ash_net.py	17.91%	55 Missing ⚠️
...uncertainty/datamodules/classification/imagenet.py	41.75%	48 Missing and 5 partials ⚠️
...uncertainty/datamodules/classification/cifar100.py	25.42%	43 Missing and 1 partial ⚠️
..._uncertainty/datamodules/classification/cifar10.py	24.56%	43 Missing ⚠️
torch_uncertainty/ood/nets/adascale_net.py	22.50%	31 Missing ⚠️
torch_uncertainty/ood/nets/scale_net.py	26.47%	25 Missing ⚠️
torch_uncertainty/ood/utils.py	45.16%	17 Missing ⚠️
... and 3 more

❌ Your patch check has failed because the patch coverage (39.02%) is below the target coverage (95.00%). You can increase the patch coverage or adjust the target coverage.
❌ Your project check has failed because the head coverage (89.89%) is below the target coverage (95.00%). You can increase the head coverage or adjust the target coverage.

Additional details and impacted files

@@            Coverage Diff             @@
##              dev     #162      +/-   ##
==========================================
- Coverage   98.19%   89.89%   -8.31%     
==========================================
  Files         167      175       +8     
  Lines        8662     9847    +1185     
  Branches     1107     1251     +144     
==========================================
+ Hits         8506     8852     +346     
- Misses        101      906     +805     
- Partials       55       89      +34     

Flag	Coverage Δ
pytest	89.89% <39.02%> (-8.31%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.