Training adapter models in a multimodal setting

[loukasilias]

Hello,

I have a model with text and images as inputs
I pass the text through BERT and images through Vision Transformers and train the multimodal model.
Is it possible to add adapter to BERT and train the multimodal model in the same way (as I train it without adapter / without changing the code of training).

Thank you.
Loukas

[hSterz]

Hi @loukasilias,

This depends on your trainings script but in theory you can add adapters to your BERT model easily with very little changes of the training (see our docs). You would need to:

• use the BertAdapterModel or call adapters.init(model).
• add and activate the relevant adapters to the model with model.add_adapters("name", "lora")(see here for the different adapter types that you can use) and model.train_adapters("name").
• use the AdapterTrainer instead of the Huggingface Trainer class during training (Depending on your trainings setup e.g. are you only training the adapters or in combination with the multimodal model, you might have to do additional adaptations).

Hope this helps!