The released pretrained models are also trained on down-stream tasks?

[luomancs]

Hi,
From the paper, https://arxiv.org/pdf/1910.10683.pdf, T5 is pretrained on corpus C4. Also there are lots experiments of different tasks, like GLUE and Question Answering. I wonder when we load the pre-trained model from hugging face, by the following code, model = T5ForConditionalgeneration.from_pretrained('t5-base'), the weights of this model is just pretrained on C4 or also pretrained on all the downstream tasks?
I found then when I evaluate "model" initialized from t5-base performs well on SQuAD (86 F1), which makes me think it has already seen SQuAD.

Thanks.

[craffel]

T5-Base is trained on a multi-task mixture including SQuAD. If you want a model trained on an unsupervised task only, use T5.1.1.