Benchmarks-Evaluation

Open Benchmarks of Zero-shot Learning (ZSL)

Content

1. Computer Vision
1.1 Image Classification
1.3 Action Recognition	1.4 Visual Quesition Answer
1.5 Image Retrieval
2. Natural Language Processing
2.1 Text Classification	2.2 Fine-grained Named Entity Typing
2.3 Relation Extraction
3. Knowledge Graph Refinement
3.1 Link Prediction with unseen relations	3.2 Link Prediction with unseen entities
4. Graph
4.1

Computer Vision

Image Classification

Dataset	Size	Granularity	Split (Seen/Unseen)	# Images	Side Information	Sources
Attribute Pascal and Yahoo (aPY)	small	coarse	20/12	15339	attribute	Paper Data
Animals with Attributes1 (AwA1)	medium	coarse	40/10	30475	attrbute	Paper Data
Animals with Attributes2 (AwA2)	medium	coarse	40/10	37322	attrbute	Paper Data
Caltech-UCSD- Birds 200-2011 (CUB)	medium	fine	150/50	11788	attribute, hierarchy	Paper Data
North America Birds (NAB)					attribute, hierarchy	Paper Data
SUN	medium	fine	645/72	14340	attribute	Paper Data
ImageNet	large	coarse & fine			class hierarchy from WordNet, word embeddings of class names	Paper Data

• The datasets of aPY, AwA1, AwA2, CUB, SUN and ImageNet have a standardization version, which is widely used now with more standardized data splits, image features and attribute vectors, and evaluation metrics.
• evaluation metrics (accuracy)

Natural Language Processing

Text Classification