About asynchronous generation and training like AReal

[llv22]

One question about ART framework, will we plan to support asynchronous generation/rollout and training, like https://github.com/inclusionAI/AReaL?tab=readme-ov-file (paper: https://arxiv.org/pdf/2505.24298)?

Essentially, it is a non-blocking rollout mechanism so that the ready-to-use rollout group can be sent to training queue, instead of waiting (If I understand correctly)

Best Regards
Orlando

[saum7800]

Hey Orlando!

We have a helper for doing Async RL. check out art.trajectory_group_batches(). example of using it can be found in

ART/dev/tau-bench/run_rl.py

Line 247 in aab795f

async for trajectory_groups in art.trajectory_group_batches(

. However, we have seen some stability issues during training due to rollouts being off policy, but i'd love to know what results you see on your tasks!

Let me know if you meant something else here

[llv22]

@saum7800 Thanks a lot for your quick response. Will try to apply async RL and let you know the result ^-^. We now run our experiment on H200, rollout/training=3/1 somehow