Shape[0] of inputs["states"] in shared model of PPO

[carlulxy]

In the PPO algorithm, the code provides a shared model to compute the action or value from the input["state"]. I have some questions about the shape of input["state]. I realized that input["state"].shape[0] = env.num_envs is not always true. Sometimes input["state"].shape[0] = 2*env.num_envs. What does this mean? Is it a continuous multi-frame state? And if input["state"].shape[0] = 2*env.num_envs, then the shape[0] of action computed is also changed to be 2*env.num_envs. Then how is this action applied to render the environment?

I found this out when I wanted to cache the multi-frame state as an input to the model.

[Toni-SM]

Hi @carlulxy

The .shape[0] value depends on the PPO algorithm execution:

• during rollouts (collecting env transitions) the shape[0] value is ensured to be env.num_envs
• during algorithm update (learning process) the shape[0] value could be env.num_envs or rollouts * env.num_envs / mini_batches (in this last case. if the division is not exact, the mini-bash may have a different shape[0] value)

[carlulxy]

Thank you for you reply！
That is, even when .shape[0] is not env.num_envs, it doesn't represent the state of multiple consecutive frames, but rather multiple states (not necessarily consecutive) extracted from Experience Replay Buffer, right?
And as i have mentioned in another discussion #342, i want use RL to optimize the diffusion model that takes the state of multiple consecutive frames as input . So I have to do the processing in get_observation of the task class, right?