Does Llama 3.2 vision use CLIP?

[vedantroy]

2 questions really:

1. In the [source code] (https://github.com/pytorch/torchtune/blob/main/torchtune/models/llama3_2_vision/_model_builders.py) for Llama 3.2 vision, there's a component called clip_vision_encoder. I assume this is not referencing OpenAI CLIP?

2. Is this a correct representation of the Llama 3.2 vision architecture? I read the paper + browsed the source code, and this is what I came up with:

flowchart LR
    vision["{image}"] --> clipT[CLIP Transformer]
    clipT --> t[t Transformer]
    t --> cat[cat along feature dim]
    
    clipT -- "hidden state 1" --> cat
    clipT -- "hidden state 2" --> cat
    clipT -- "hidden state 3" --> cat
    
    cat --> MLP
    
    MLP -- "cross attention" --> decoder[Decoder]
    text["{text}"] --> decoder

Loading

[pbontrager]

1. The base vision encoder in Llama3.2 is a variant of OpenAI CLIP. So we implement the generic model and then in the Llama3.2 encoder builder use the hyperparameters used by the Llama team and not the original CLIP model. The Llama CLIP is pre-trained in the same way as OpenAI CLIP and then further post trained with end to end with the encoder head and decoder.

2. The diagram looks good where I'm assuming that "t Transformer" is the transformer inside of Llama3VisionProjectionHead. Also the cross attention with the decoder only happens on certain layers of the decoder. If you want to dig even deeper into how it works you can look at how images get broken into tiles which CLIP converts into patches so that you end up with a encoder sequence length of num_imagesxnum_tilesx(num_patches**2 + 1)

[sovit-123]

@vedantroy As I understood from the Llama 3 paper (around page 56). The authors took a version of the original ViT Model (vanilla vision transformers). Then trained it with image-text pair again for text supervision and that encoder was used for the multimodal Llama 3.