Animatable avatars for MuJoCo models in Unity

[Balint-H]

To be able to apply animations in the Unity plugin, or exploit all the nice rigging methods implemented in the engine for kinematic controllers, a model needs to be imported in an FBX format. To have a 1:1 equivalence with the MuJoCo model, it is important that they have matching local positions of body parts. To achieve this you can remove all Mj components from your humanoid model, and can then use the FBX exporter package in unity to create an animatable rig, which you can reimport.

During the import process, if its a humanoid rig, you can select it as such. In this case you will probably have to edit the skeleton, and reassign a few of the transforms to the correct body-parts. Once done, you can track the individual transforms with MjMocapBodies to get their information to the simulation (or to connect regular bodies to them with constraints).

Here's an example I've been working with for the humanoid model from (Merel et al. 2017):
humanoid_CMU.zip

If someone has a recommendation on how to combine the individual meshes into a single weighted mesh for neater material assignment that doesn't involve manually recreating the skeleton and painting the vertex weights, let me know. For the time being I've been using an editor script to apply a material to all meshrenderers of the animated model.

It's also possible to do it with the traditional MuJoCo humanoid (e.g. Tassa et al. 2012), but for humanoid rigs in Unity it is important to have the character in a T-pose. To achieve this, you can modify the local transform of body segments, but don't offset the Geom objects themselves.

[shubhankarKulk]

Can we convert fbx animations so that they can be replayed on humanoid models in MuJoCo?

[Balint-H]

Mismatches between animation hierarchies is why you'd need retargeting and IK. The specific script from dm_control I sent doesnt actually use physics (if I remember it correctly), rather just converts kinematics (animation) to the state space of the MuJoCo model so it can be played back.

For your specific problem a manual solution could be adding the missing dofs and parent root body to the mj model, or processing the animation so the animation root is the pelvis.

[shubhankarKulk]

I created a model which is has a root body on the pelvis and added the rest of the limbs accordingly. Additionally, I am trying to get how to convert the fbx motion capture data to a mujoco humanoid, but I am encountering problems with the rotations. When I load the fbx data in blender, there is a 90 degree rotation on the x axis. I tried to correct this rotation in mujoco, but have been unsuccessful. Any help would be appreciated. The animations I am trying to use are default mixamo fbx animation, as you can see in the actualMotionCapture video, and the improvedImplementation video is the one I have tried to implement.
actualMotionCapture.webm
improvedImplementation.webm

[Balint-H]

Converting motion file formats to one another can be really tricky, I've banged my head against similar problems in the past 😄

Once you load the anim in blender how do you continue to process it?
I've found that BVH is a bit easier to interpret than FBX formats, so you may want to export to that and work with its serial euler angles format.

[shubhankarKulk]

That is interesting. I usually converted the fbx format to a text document containing positions and orientations (quaternions). I simply remapped these values to the qpos values of the humanoid that I created. It gives somewhat of a promising result. I will have a look into bvh file format, as I believe there is a blender script available for direct conversion of fbx to bvh file formats. I am also unable to understand how bvh files are structured. Could you help me out in this regard?

[Balint-H]

The BVH exporter is built into blender which is quite convenient. Just select your animated rig and export it under File. Here's a nice breakdown on BVH format: https://staffwww.dcs.shef.ac.uk/people/S.Maddock/publications/Motion%20Capture%20File%20Formats%20Explained.pdf