Various Training Results So Far

[afiaka87]

Edit: I'll work on getting the parameters for each of those inlined, but in the meantime if you're desperate and understand wandb.ai, you should be able to find the exact train_dalle.py that I used by clicking on the run name and going to the Files tab on the left.

I trained dalle-pytorch on a couple different datasets a few times. How "successful" the results are is up to you, but one thing I learned is that we need a lot more image-text pairs to get to the level of accuracy that OpenAI has.

~1 Epoch on COCO, OpenImage and OpenAI blog post image-text pairs I scraped.

On this one, I think the OpenAI image-text pairs got overfit and did more harm than good to the generalizability of the model
https://wandb.ai/afiaka87/OpenImagesV6/reports/Training-on-COCO-OpenImage-Blogpost--Vmlldzo1NDE3NjU

~1 Epoch on COCO2014

https://wandb.ai/afiaka87/OpenImagesV6/reports/OpenImagesV6-COCO--Vmlldzo1MzgyNTI

~1 Epoch on the OpenAI blog post image-text pairs I scraped.

https://wandb.ai/afiaka87/ellie_miller/reports/Project-Dashboard--Vmlldzo1NDQ5MzM

~1 Epoch on COCO and OpenImagesV6 (no blog post images)

https://wandb.ai/afiaka87/oai_coco/reports/Trained-on-COCO-2014-and-OpenImagesV6--Vmlldzo1NDQ5NDg

Roughly 3 full epochs OpenAIBlog Scrape and COCO2014

This one crashed during its first run, so is split into two reports. It uses a larger batch_size of 48, so it needs fewer iterations than the other runs to finish an epoch.
https://wandb.ai/afiaka87/openai_blog/reports/OpenAIBlog-Scrape-and-COCO2014-part-1---Vmlldzo1NDQ5NjU

https://wandb.ai/afiaka87/openai_blog/reports/OpenAIBlog-Scrape-and-COCO2014--Vmlldzo1NDQ5NTQ

(Colab and an A100) Four Runs on COCO2014 (200k images) ranging from 5 to 8 epochs trained.

The 1024 token model allows us to train in colab! Two runs are P100s. One run is a V100, and finally I also trained on an A100 i had access to.

https://wandb.ai/afiaka87/dalle_train_coco2018/reports/Various-Runs-on-COCO2014--Vmlldzo1NDQ5NzY

[afiaka87]

A few of these have a dalle.pt checkpoint file. You'll need to click on the project name at the top of the screen, find artifacts on the left side, and then download it if you would like to begin training from there. None of the checkpoints are particularly worthwhile yet however, so I dont see much need for this.