Access to 5-shot training data for rare-species dataset

[emazuh]

Hello, I am looking to compare some on-device model performance for the rare-species dataset by end of week (9/19/2025) for a paper submission and would like to cite the bioclip2 few shot accuracy values. However, I am unable to find the seen_in_training data online although "Five-Shot Classification" is mentioned on the huggingface data page. They don't seem to overlap with the official huggingface dataset (using load_dataset) base file names or the manual download dataset (using Dataset Structure step 3 download instructions) base file names.

My current plan is to randomly sample five items from the test dataset for training and report few shot test performance on the rest of the data but this can't be compared with the bioclip2 paper values for reference. Any assistance in finding the five shot training data would be greatly appreciated! @rwightman @rom1504 @mitchellnw @Zasder3 @samuelstevens @carlini

[egrace479]

For the original BioCLIP model, seen_in_training vs unseen_in_training was in reference to the species themselves.

For BioCLIP 2, we did not completely remove the species from training. The Rare Species dataset images were not included in training, allowing them to provide a valid benchmark. However, the species were likely seen in training, just through images from other sources.

[emazuh]

Thanks for the quick response @egrace479 ! Here is my understanding so far (kindly correct me i I'm off). Since the training data I would use for this dataset will have to be a subset of the same (rare-species), I can't actually compare to CLIP 1 or 2 (1 sees some species and evaluates on other unseens ones, 2 might see some instances from any of the species but not directly reported). The way forward is to train on a small subset of the dataset and report on the rest but compare with performance from other small models trained in a similar manner.

[samuelstevens]

I think the best way to evaluate BioCLIP and BioCLIP 2's few-shot accuracy on the rare-species task is to:

1. Randomly sample 5 images from each species as your training split
2. Fit a SimpleShotClassifier (I wrote one that has an sklearn-like API here: https://github.com/samuelstevens/biobench/blob/main/biobench/simpleshot.py) to the training split
3. Evaluate on the remaining images, measuring micro accuracy.
4. Repeat this process 3-10 times, re-sampling your training split each time.
5. Report mean micro accuracy and std dev micro accuracy.