Unit 4. Multimodal models. Transfer Learning outline

[minemile]

Hi all!
We have slightly updated our outline for the multimodal model transfer learing chapter. We would be glad to receive any feedback ;)

1. Introduction (@minemile)

• A brief introduction to how multimodal models can be used for a specific task.: one-shot, few-shot, train from scratch and transfer learning. Next, we will talk in more detail about transfer learning (definition, advantages, etc). We will also look at practical transfer learning applications using examples in jupyter notebooks.

2. Jupyter notebooks

• Fine-tune CLIP (@minemile)
• Visual Question Answering (@cfalholt)
• Image-to-Text (@Zekrom-7780)
• Open-set object detection (@minemile)
• Assistant (GPT-4V like) (@cfalholt)

3. Conclusion (@cfalholt)

• Compare and discuss results of previous sections.
• Discussion on the benefits and challenges of each method.
• Insights into future developments within the field.

Plan for each notebook:

1. Introduction.
   • Task description
   • Dataset description
2. Full fine-tuning (if possible).
3. PEFT training (if possible)
4. Inference result discussion
5. Conclusion

Team members - @minemile, @cfalholt, @Zekrom-7780
bcc - @johko @lunarflu @merveenoyan

[husnain-ali21]

Hello @minemile @cfalholt I would like to contribute on this chapter.

[minemile]

Hello! Great! You can join our discord channel and we will discuss in more details.

[gonzachiar]

Hi! I also sign up myself to contribute here. Can you share the discord channel again @minemile?

[minemile]

Hi! Sure: https://discord.gg/9HnbbDq5

[ATaylorAerospace]

This outline looks very comprehensive. @cfalholt how detailed were you planning to make the Jupyter notebook for Assistant (GPT-4V like)?