Time to release show-o2 video understanding and generation model

[Telephone1024]

Very nice job， may I ask when you plan to release show-o2 video understanding and generation model？

[Sierkinhane]

Hi, thank you for your interest in our work. We are actively refining the models for video generation, and their release depends on the company's inspection process, as we utilize internal data. Specifically, we do not incorporate video understanding data into our training pipeline; however, a straightforward solution involves using multiple frames for video comprehension. We will release the training and inference code for single image-text, interleaved image-text, and video data as soon as possible. I believe this will assist in fine-tuning our models for downstream tasks like video understanding.

[zilaiz]

Thank you for the great work! A related follow-up question is whether the current released show-o2 model checkpoints (1.5B and 7B) are pre-trained on interleaved image-text data and if they are functionally ready (maybe not perfect) for interleaved generation during inference. For example, if prompted correctly, do those models support automatic switch from text generation to image generation by predicting the [BOI] token?

[Sierkinhane]

Hi, models supporting mixed-modality generation will be released in the coming weeks (hopefully), as I'm currently doing instruction tuning and making it accommodate more images in one sequence. Or we may release a simple version of this tuned only on a small dataset, like the visual storytelling dataset.

In the mixed-modality generation mode, the model will predict the [BOI] token when it wants to generate an image.