Latent generation in VERL RL stage

[innsn]

Hi,
Thanks for the cool work!

I’d like to discuss the RL part after the two-stage training. I noticed that you implemented latent generation by modifying the generate part in the Transformers. In VERL, I’m wondering how this is achieved — is the rollout generation set to HF Transformers? This approach seems to consume a lot of GPU memory.

Looking forward to your reply.

[Unicorn53547]

Hi,

Sorry for the late response.

Yes—we set the rollout generation to HF Transformers and have modified both VERL and Transformers for RL training.

Indeed GPU memory is the main bottleneck. With smaller images (typically ≤256 patches), we can train RL and see some improvements; but with high-resolution inputs (e.g., visual-search images with more than 1k patches), memory usage becomes a problem for us.

We’re working on the RL training (especially the memory issue) before releasing this part of the code. If you have any suggestions or recommended implementations, we’d love to hear them!

[HenryHZY]

Hi, I am not very familiar with VERL. Could you share some tips about the reproduction of RL training for Mirage?