Rig disjoint like a bone pile when lost hand tracking

[conceptfac]

When the player hands are not visible, the character skeleton dismount like a pile of bones.

[andkim-meta]

Is the issue occurring when the player's hands (mesh) in the app isn't visible due to lost hand tracking, or does it still occur if the mesh is visible? If it's the former, as the current retargeting logic depends on Unity's animation system, when the GameObject is no longer being skinned the skeleton will not render correctly. This would be the case if this issue only occurs with the in-app hands mesh not being visible in the current user's view. If it's the case and the skinning issues when the mesh is not visible on screen is the cause, we recommend using the SkinnedMeshRenderer.updateWhenOffscreen API (https://docs.unity3d.com/ScriptReference/SkinnedMeshRenderer-updateWhenOffscreen.html) or changing the bounding volume of the character with the skinned mesh renderer (https://docs.unity3d.com/Manual/class-SkinnedMeshRenderer.html).

[JordiBlancoDev]

I'm having the same problem but without using a rig, i have a bunch of spheres follow each OVRBone the OVRSkeleton returns from OVRSkeleton.Bones, it starts tracking well but after some time it dismounts like a pile of bones.
This is what im doing to track the bones position:

foreach(OVRBone l_Bone in l_OVRSkeleton.Bones)
{
	GameObject l_VisualBone=GameObject.CreatePrimitive(PrimitiveType.Sphere);
	l_VisualBone.name=l_Bone.Id.ToString();
	l_VisualBone.transform.localScale=Vector3.one*0.025f;
	l_VisualBone.transform.parent=l_Bone.Transform;
	l_VisualBone.transform.localPosition=Vector3.zero;
}

This happens both with full-body and upper-body tracking, and with high and low fidelity, the full code is as below

async void StartBodyTracking()
{
	OVRSkeleton l_OVRSkeleton=_OVRSkeleton.GetComponent<OVRSkeleton>();
	OVRPlugin.StartBodyTracking2(OVRPlugin.BodyJointSet.FullBody);
	OVRBody.Fidelity=OVRPlugin.BodyTrackingFidelity2.High;
	OVRPlugin.RequestBodyTrackingFidelity(OVRPlugin.BodyTrackingFidelity2.High);
	while(l_OVRSkeleton.Bones.Count==0)
		await Task.Yield();
	foreach(OVRBone l_Bone in l_OVRSkeleton.Bones)
	{
		GameObject l_VisualBone=GameObject.CreatePrimitive(PrimitiveType.Sphere);
		l_VisualBone.name=l_Bone.Id.ToString();
		l_VisualBone.transform.localScale=Vector3.one*0.025f;
		l_VisualBone.transform.parent=l_Bone.Transform;
		l_VisualBone.transform.localPosition=Vector3.zero;
	}
}

[sohailshafiiWk]

Hello, please try using OVRBody.cs to start body tracking and set the fidelity level instead of writing code to handle those cases. Furthermore, OVRSkeleton.cs has fields that indicate if the data has initialized (IsInitialized) and has been validated (IsDataValid). We recommend that you use those APIs to make sure that you are not using bone data if those fields are false.

If you run into the scenario when OVRSkeleton.cs indicates valid data and the bones dismount into a pile, please reply with how you are able to reproduce that specific bug.

[JordiBlancoDev]

Hello, please try using OVRBody.cs to start body tracking and set the fidelity level instead of writing code to handle those cases. Furthermore, OVRSkeleton.cs has fields that indicate if the data has initialized (IsInitialized) and has been validated (IsDataValid). We recommend that you use those APIs to make sure that you are not using bone data if those fields are false.

If you run into the scenario when OVRSkeleton.cs indicates valid data and the bones dismount into a pile, please reply with how you are able to reproduce that specific bug.

Hello, I've tried what you told me with no luck, but i discovered something interesting, no matter what setup i have (initializing through my script or normally), if i have passthrough enable, it will always dismount into a pile randomly (usually when i move my arms or head), but if passthrough is disabled, it never dismounts, any idea why could this be happening?
All i have is a scene with a gameobject containing:

• OVRCameraRig
• OVRManager (with permission for body tracking and passthrough set to required and enabled)
• OVRSkeleton, OVRBody (both set to Full Body)

if to this i add the OVRPassthroughLayer (set to underlay) the body tracking starts failing, but if i remove the component it'll work fine

[sohailshafiiWk]

if i have passthrough enable, it will always dismount into a pile randomly (usually when i move my arms or head), but if passthrough is disabled, it never dismounts, any idea why could this be happening?
All i have is a scene with a gameobject containing:

Hello, can you tell us which device you are running, and which OS version? The latter can be found in the "Software Update" section of the Device's OS settings.

[JordiBlancoDev]

Im using Meta Quest 3, i can't check the OS version right now but im pretty sure it was the latest update

[MJ35512]

I also have the same problem. When I lose hand tracking my avatar becomes a bag of bones.
I use a Quest3 v74, Unity 2022.3.36f1, Unity-Movement v72.0.0 and Ready Player Me which takes care of bones retargeting.
The problem seems to be linked to a need for computing power (difficult to reproduce in Link mode with a powerfull PC, on Quest 3 less common without passthrough)
I work around the problem by destroying all the objetcs and components created for the retargeting and resetting it, which is not a good solution on my side. Indeed, I do not know how to automatically detect this problem so I reset the avatar upon user action...

[sohailshafiiWk]

Im using Meta Quest 3, i can't check the OS version right now but im pretty sure it was the latest update

Can you try to reproduce this issue by adding passthrough to our retargeting scene? If that happens, can you share the project (without the Library, Temp or unnecessary items in Assets) with just that scene for us to debug?

[JordiBlancoDev]

I've added OVRPassthroughLayer component to the OVRCameraRig in the retargeting scene and it's still happening. Don't know if i mentioned that this only happens in build, in editor i could not make passthrough work so i couldn't test it directly.
Link to the project: https://we.tl/t-jq4QoSEGs4

[conceptfac]

I just discovered one thing, this only happens using Passthrough here!

[MJ35512]

Im using Meta Quest 3, i can't check the OS version right now but im pretty sure it was the latest update

Can you try to reproduce this issue by adding passthrough to our retargeting scene? If that happens, can you share the project (without the Library, Temp or unnecessary items in Assets) with just that scene for us to debug?

It was difficult to reproduce but I succeeded.
Context : Quest3 v74, Unity 2022.3.36f1, Unity-Movement v72.0.0
I just used the Body Tracking Samples with the MovementHighFidelity scene (with passthrough) deployed on the headset.
To reproduce the bug, I hid my hands under my desk and moved my head slightly.

I also reproduce the bug with the MovementRetargeting scene (with passthrough) deployed on the headset.

But I never succeeded to reproduce the bug without passthrough.

[JordiBlancoDev]

I've done a little bit more of digging and i believe it has to do with the CPU Level and its cap at level 3 when passthrough is enabled, because every time i disabled passthrough the CPU Level jumped to 4 and it worked perfect, but when i enabled passthrough the CPU Level went down to 3 and it broke.
Is there no way to force a higher CPU Level than 3 with passthrough enabled?