Randomize articulation joints in replicator

[cowcnovo]

Is it possible to randomize the joint positions of a robot arm, or any articulation, use the replicator API? I have tried to modify the "position" attribute of my robot but I get the error that there is no such attribute. Here is my attempt:

for joint in joint_paths:
    with rep.get.prim_at_path(f"/Replicator/Ref_Xform/Ref{joint}"):
        rep.modify.attribute("position", rep.distribution.uniform(0, 180))

I have noticed that there isn't really much on this topic of incorporating an articulation into replicator scripts.

[zwdoescode]

Here is a documentation about replicator randomizer https://docs.omniverse.nvidia.com/extensions/latest/ext_replicator/randomizer_details.html#randomizer-examples.
There is an example to randomize position, rotation and scale of an object.

I don't see an example to randomize joint positions. Let me reach out to the internal team to see if that is possible

[cowcnovo]

Hi @zwdoescode,

Thank you! I hope this is possible as it is a very important feature which is appropriate in synthetic data generation! Hope to hear from you soon.

[liilatorabi]

Can you explain what is your use case? Are you using Isaac Sim or Isaac Lab?

[cowcnovo]

Hi @liilatorabi,

My idea:
To simulate a robot arm picking and placing objects. The scene consists of a table on which there are cubes, cylinders and cones and a UR robot. The camera will use a trained object detection model to estimate the object positions.

To train the model, I am using Isaac Sim Replicator. I have been able to vary essentially everything that is needed apart from the robot's joints. I need to vary the robot joint positions since the camera looks slightly towards the robot (along with the objects of course) and it would be nice to have random joint configurations for the robot in the background to make the vision model more reliable.

To simulate the robot, and all of its pick-place logic, as well as integrating this trained computer vision model, I am using Isaac Lab. But, the only reason for this is because I wanted to make the UR robot learn to pick and place. Isaac Sim can also be used to simulate here instead.

What works:

1. I have been able to train an object detection model based on the replicator's synthetic data generation
2. I have deployed this model on Isaac Lab as observations to the agent

TLDR:
I just need to know how to vary joint angles in replicator so that the camera/vision model does not always see the same stationary robot since while pick-placing, the robot will indeed move.

[liilatorabi]

Thanks for explaining. This should be fairly straight forward.
@ahaidu-nv do you have any hints?

[ahaidu-nv]

@cowcnovo I am not aware of any Replicator OmniGraph node approach to randomize the articulation joints, maybe the generic node of randomizing attributes (rep.modify.attribute) might work:

    with lights:
        rep.modify.attribute("inputs:color", rep.distribution.normal((0.2, 0.2, 0.2), (1.0, 1.0, 1.0)))

Otherwise you can use the isaac sim randomization OmniGraph node for articulations:

• https://github.com/isaac-sim/IsaacSim/blob/main/source/extensions/isaacsim.replicator.domain_randomization/python/nodes/OgnWritePhysicsArticulationView.py
• https://github.com/isaac-sim/IsaacSim/blob/main/source/extensions/isaacsim.replicator.domain_randomization/python/tests/test_ogn_write_physics_articulation_view.py

Or the most straightforward aproach would be to use Isaac Sim / USD APIs to randomly modify the joint attributes whenever needed, and to use replicator for writing the data only. Basically separating the randomiztion from the data collection.

Here are some basic getting-started snippets with this approach:

• https://docs.isaacsim.omniverse.nvidia.com/latest/replicator_tutorials/tutorial_replicator_getting_started.html