Skip to content

TRL Updated version of VLM GRPO update along with GSPO #3132

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 39 commits into from
Sep 16, 2025
Merged

TRL Updated version of VLM GRPO update along with GSPO #3132

merged 39 commits into from
Sep 16, 2025

Conversation

@pluesclues
Copy link
Collaborator

@pluesclues pluesclues commented Aug 12, 2025 •
edited
Loading

Original PR and credits to #2752 and @GAD-cell. Compared to the original PR, this just keeps in mind that TRL supports VLM GRPO natively. This PR also adds GSPO and depends on unslothai/unsloth-zoo#233.

pluesclues added 21 commits June 22, 2025 20:59
@pluesclues
Kept, padding logic
f911c32
@pluesclues
Made sure prediction step in rl.py allows logging for callbacks in RL…
2ba7f50 
… trainers
@pluesclues
Merge branch 'unslothai:main' into main
0c1bc4d
@pluesclues
updated llama.py to new online_dpo changes
78336ce
@pluesclues
Update rl.py to make logic simpiler
383aa9c
@pluesclues
Update rl.py, made sure tokenized_output on eval step was on same device
532af4f
@pluesclues
Update rl.py, corrected tokenized_outputs to inputs
49f77c1
@pluesclues
Update rl.py, removed sagemaker stuff
7921aa7
@pluesclues
Update llama.py, figures out if there is right padding automatically
54f03ee
@pluesclues
Update llama.py, changed conditional statement for right padding slig…
a8d4168 
…htlyt
@pluesclues
Update llama.py, updated OS.environ variable to temp variable
236b924
@pluesclues
Merge branch 'main' into main
76d73c6
@pluesclues
Update rl.py, made it account for right padding in online dpo and rew…
fa2e18e 
…ard modeling
@pluesclues
Update llama.py, automatically figures out if right padding is needed
80f9cd2
@pluesclues
Merge branch 'main' into main
ed1771a
@pluesclues
Merge branch 'main' into main
49d3844
@pluesclues
Merge branch 'unslothai:main' into main
b0a9c65
@pluesclues
Update rl_replacements.py, fixed up passing image data to functions
30f3366
@pluesclues
Merge branch 'unslothai:main' into vlm_grpo_update
327053f
@pluesclues
Update rl_replacements.py, for VLM GRPO support with TRL
8af680f
@pluesclues
Update rl_replacements.py, gspo added
5e0fbdb
@pluesclues pluesclues mentioned this pull request Aug 12, 2025
Merged
Datta0
Datta0 reviewed Aug 23, 2025
View reviewed changes
@GAD-cell GAD-cell mentioned this pull request Sep 1, 2025
Open
Datta0
Datta0 reviewed Sep 5, 2025
View reviewed changes
@pluesclues pluesclues mentioned this pull request Sep 9, 2025
Closed
@pluesclues pluesclues mentioned this pull request Sep 16, 2025
Merged
danielhanchen
danielhanchen reviewed Sep 16, 2025
View reviewed changes
danielhanchen
danielhanchen reviewed Sep 16, 2025
View reviewed changes
danielhanchen
danielhanchen requested changes Sep 16, 2025
View reviewed changes
Copy link
Contributor

@danielhanchen danielhanchen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice

@danielhanchen danielhanchen merged commit 96c2ecb into unslothai:main Sep 16, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Datta0 Datta0 Datta0 left review comments

@danielhanchen danielhanchen danielhanchen requested changes

Assignees

@Erland366 Erland366

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@pluesclues @danielhanchen @Datta0 @Erland366