Outline

[pedrogengo]

Introduction

• History of multimodal models @mattmdjaga
• Pre-clip @mattmdjaga
• Post clip @mattmdjaga

Losses/ self supervised learning

• Contrastive @pedrogengo
• Non contrastive @pedrogengo
• Triplet @zaristei3
• One or two other ones @zaristei3

Models

• CLIP (Image & text embedding) @pedrogengo
  • Introduction
  • Theory behind clip
    • Variations of CLIP backbones
    • How tokenisation and embeddings work in clip
  • Applications
  • Finetuned versions
• Donut or Nougat (Document Analysis)
• GroupViT or OneFormer (Segmentation) @zaristei3
• BLIP (Vision language with text generation)
• OWL-VIT (Vision Language object) @mattmdjaga

Practical applications & challenges

• Challenges:

  • Data bias/ out of distribution data
  • Hard to get enough data -> leads to using noisy internet data

• Possible Practical Notebooks

  • Search image engine based on textual prompts
  • Downstream tasks on embeddings eg classification, clustering etc
  • Visual question answering systems

[ATaylorAerospace]

@pedrogengo For the Finetuned Models under "Finetuned Versions" were you planning to focus on CLIP for General Purpose pretrains like Falcon or XGen or focus on Image/ Video Related pretrains like Giraff-70B or Video-ViT-Base-32?

[SkalskiP]

Hi 👋🏻 Is there any chance I could pick any of those?

• Search image engine based on textual prompts
• Downstream tasks on embeddings eg classification, clustering etc

[pedrogengo]

Hi! We will have someone in the team that will work on these notebooks :(. However, if you want to contribute to another notebook, feel free to suggest! We are more than open to have new contributors