MyoSuite ==> Blender-Unity ==> AR/VR

[vikashplus]

Platforms such as Unity, Blender, etc. are mature ecosystems supporting diverse graphics and simulation needs.

Opening a call to building MyoSuite - Unity/Blender interfaces that can help explore needs around model design, rendering, packaging, AR/VR applications, etc.

Example ideas :

• Define the basic Unity-MyoSuite interface by 1) Importing mujoco into Unity and 2) exposing the Unity-MyoSuite environment (ml-agents)

• Port models and tasks in Unity. Reproduce and match between MyoSuite <> Unity-MyoSuite

• Integrate MyoSuite and explore uscases with VR platforms e.g. Oculus, VisionPro, etc

[Vittorio-Caggiano]

cc: @Balint-H @ ttktjmt

[Balint-H]

Environments that could benefit the most would be locomotion environments for Unity's animation pipelines, and reach+manipulation which could use visual observations relying on shading and assets for Unity.

What do you see as potential VR scenes that use MSk environments? I get the angle of simulating VR users to get estimates of fatigue and comfort, but what about applications with a real VR user in the loop?

[vikashplus]

1. Motion imitation is turning out to be a high-value target. We can envision the target being streamed in real time from the VR user for self imitation.
2. We can also think of collaborative task with the policy
3. There are multiple rehabilitative strategies that follow the leader-follower paradigm, which canbe interesting to look at.

[PatP15]

If there is need for assistance for motion imitation in the Unity pipeline, please reach out to me! Would love to assist with the project.

[vikashplus]

@Balint-H

[Balint-H]

@PatP15, a fork of https://github.com/Balint-H/modular-agents could be a good starting point! It's locomotion with imitation learning project, with MuJoCo in Unity, but not with MSk. We'd need a full-body MSk skeleton for that, could you remind me if that has been released @vikashplus?

If you do want to start making a MSk variant of those environments, let me know or send issues/discussion to the Modular-Agents repo!

[vikashplus]

I'd recommend starting simple. Here are the two options I'd recommend

1. Building a locomotion agent using the myolegs_abdomen model. The winning entries from chase-tag myochallenge inspire this suggestion
2. Building full-body controllers for MyoSekelton in the joint space.

Once we have signs of life in these reduced domins, we can strategize scaling up from there.