Deploying policy to from Isaac Lab to Isaac Sim

[PanagiotisKotsaris]

Hello,

I have trained a robot in Isaac Lab, using the rsl_rl library, to reach some targets and avoid collisions using a depth camera. When I play the environment, the robot seems to behave as expected (reaches target and avoids collision). I am trying deploy the policy (.pt file) in Isaac lab and the robot does not reach the targets. It moves in unexpected ways and gets stuck on obstacles.

The issue is that as far as I can see I have used the same observations in Isaac Sim as I have used in training. The observations are very simple, they are:

• mdp.joint_pos_rel
• mdp.joint_vel_rel
• mdp.last_action
• mdp.image
• target_ee_position (custom observations that just gets the target position from the command manager)
• current_ee_pos (custom observations that just gets the current position of "link 6 " from the RigidObject data)

The action is scaled to 0.5 - same as in training - and the robot is from the same USD file as in training. The gains are the same - they are the ones defined in the USD, no custom values. I could not find any other suggestions in the article .

Can you give me any other suggestions?

[RandomOakForest]

Deploying a trained RL policy from Isaac Lab to Isaac Sim can encounter several issues even when observations and actions appear consistent. Here are some considerations and solutions:

1. Policy Export and Normalization

• Observation Normalization: Policies trained with RSL-RL often expect normalized inputs. Ensure you apply the same normalization statistics (mean/std) used during training. These are typically stored in agent.yaml and env.yaml files generated during export.
  • Example code for normalization:

    # Load normalization parameters from YAML files
    obs_mean = torch.tensor(normalization_data["obs_mean"], device="cuda:0")
    obs_std = torch.tensor(normalization_data["obs_std"], device="cuda:0")
    normalized_obs = (raw_obs - obs_mean) / obs_std

• Export Method: Use Isaac Lab's play.py script to export policies, which automatically generates required config files. Avoid manual torch.load of .pt files, as this skips critical preprocessing steps (see also).

2. Environment Configuration Mismatches

• Scene Differences: Verify the USD environment in Isaac Sim matches the training setup, including:
  • Physics properties (gravity, friction)
  • Collision meshes
  • Terrain parameters (even if using the same USD, check for unintended modifications)
• Device Configuration: Explicitly set the simulation device (CPU/GPU) to match training:

  # In Isaac Sim deployment script
  env_cfg.sim.device = args_cli.device  # Ensure this matches policy's device

3. Action Scaling and Clamping

• Action Space Handling: While actions are scaled to 0.5, confirm they are clamped to the expected range during deployment:

  actions = torch.clamp(actions, -0.5, 0.5)  # Match training action bounds

• Delay in Action Application: Ensure actions are applied at the same frequency as during training. Use simulation_step callbacks to synchronize policy inference with physics steps.

[PanagiotisKotsaris]

Thank you so much for detailed pointers!

I want to give a bit more information about my deployment process.

Additional information

1. I used the play script to export the JIT policy (the .pt file I am using).
2. I have checked the physics parameters and they are the same (same gravity and friction, same collision meshes and a simple flat terrain in both scenarios)
3. The action configuration I use is the following

   joint_pos_action = mdp.JointPositionActionCfg(
       asset_name="robot",
       joint_names=[".*"],
       scale=0.5,
       use_default_offset=False,
       preserve_order=True,
   )

As you can see there is no clipping defined by me and in the env.yaml file I only see this

clip: null

4. My environment is initialized using the World class from Isaac core and is initialized as follows:

self._world = World(
            stage_units_in_meters=1.0,
            physics_dt=self.physics_dt,
            rendering_dt=self.physics_dt * self.decimation,
            device="cuda:0",
            backend="torch",
        )

As you can see the physics and rendering steps are coming from variables that have the same values used in training. Also the device is the same as in training (the GPU).

5. Regarding the observation normalization, I am not sure on where this normalization happens. I am using empirical_normalization = True and the policy code that I got using the self._model.code variable after self._model = torch.jit.load(args.model_path).cuda() contains the section:

  normalizer = self.normalizer
  norm_proprio = (normalizer).forward(proprio, )

6. I have added a physics callback function that runs every 4 steps (decimation in training is 4) so I think I have covered the policy update frequency part

So I think that the normalizer is included in the policy.

Summing up

To sum up, my main questions are about normalization and clipping:

• Is normalization happening when I am calling the policy or should I do any extra steps?
• I have not set any clipping to the actions during training. Is there any hidden "safety" clamping going on?
• Are there any other common pitfalls that I should be aware of?

[ogiel]

Hey @RandomOakForest,

I feel like I have a similar case as the original poster of this git issue but with sim2real instead.

You're saying "Avoid manual torch.load of .pt files, as this skips critical preprocessing steps" which I believe, however, how should I load my RSL RL policies instead?

What I am doing is simply put like this: trained through RSL RL > export policy through play.py > load policy in custom .py file

In the .py I have:
self.policy = torch.jit.load(policy_path, map_location="cpu")

Where the policy_path ends with "/exported/policy.pt"

Some context:
What I'm doing is building a rotary inverted pendulum wiht a RL policy. In sim and play the pendulum is perfectly balanced at 180 degrees. When I insert fake state values in the policy (e.g. pendulum at 175 degrees vs 180 degrees with all other inputs (velocities etc.) at 0) the policy will give a greater action at 180 degrees compared to at 175 degrees which clearly shows at least something is wrong because motor action at 180 degrees without pendulum velocity should be (near) 0.

Any guidance or sources on how to load trained RSL RL policies for sim2real is hugely appreciated.

[Marsak24]

Hello
Have you figured it out? I have similar problem with deploying policy from lab to sim !