Questionnaire: "runner.py" can't support "PPO_RNN"?

[cold-young]

Hi @Toni-SM,
Thank you for your RL library.

I want to train PPO with LSTM model on Isaac Sim + Isaac Lab, but it may not support PPO with RNN.

I am using skrl version 1.3.0. In "utils/runner/torch/runner.py" it can't provide PPO_RNN

If I want to use PPO_RNN with Isaac Lab and a yaml file, do I need to make a wrapper for runner?

Or, implementing local agents is better than using "runner.py"?

[Toni-SM]

Hi @cold-young

Currently yaml configuration integrated Isaac Lab for skrl doesn't support RNN (there is a plan to support in next skrl releases).
However, you can use skrl's standalone script to train/evaluate on Isaac Lab as shown in skrl docs examples for NVIDIA Isaac Lab).

For RNN, you can refer to skrl docs examples for Gymnasium / Gym environments and copy the Model implementation to your custom scripts.

[cold-young]

Thanks for your advice! 😀
I have already added PPO_RNN with LSTM using skrl examples for Isaac Lab.
I just wander that PPO_RNN supports LSTM, RNN based architectures, but can this be able to support attention or transformer? If not, how can I use transformer based architecture?

Summary:

• Does PPO_RNN support transformer based architecture? if not, how can I add these models?

[Toni-SM]

Hi @cold-young

I don't have any example using attention or transformer at the moment.
But, if those layers require handling internal states (like RNN, LSTM or GRU), skrl should be prepared for it.

If you have or know about any RL example using transformer-based architecture, I can help with the migration to skrl.

[cold-young]

Thank you, @Toni-SM! 😄
If I want to use transformer on skrl, I need to add modules similar to ppo_rnn.py?(about RL policy update, compute loss).. and ppo_transformer.py (about policy and value classes) like ppo.py?

[Toni-SM]

Hi @cold-young

Currently, the RNN implementation is in a separated file (ppo_rnn.py) in order to have a clean PPO implementation (ppo.py).
So, if the transformer implementation requires extra lines on top of the current implementations, it is recomendable to create a new file for it (ppo_transformer.py)

[ivanpanshin]

@cold-young Hi :)

May I ask you to share your LSTM-based script? I'm currently having similar issues, but it seems you already managed to do what I couldn't.