TorchTitan for Multimodal models such as Qwen 2.5 VL or InternVL2

[anindya-saha]

Is there any plan to extend TorchTitan beyond LLM to VLM or Multimodal models such as Qwen 2.5 VL or InternVL2 ? Would there be any ideas from the community where to start from ?

[tianyu-l]

Thanks for your interest.

We have plans to extend to multimodal but have had limited bandwidth.

In terms of community contribution, I'll share contributing guidelines for adding new models soon.

Meanwhile, for each model to be added in torchtitan, there should be clear reasons why adding this particular model (e.g. is it SotA among its category at the moment?). If you are interested in working on this, a short proposal would be helpful.