Apple Vision Pro Need Face Tracking

[blacktheon]

I would be great if we support face tracking by asking the Persona right from VisionOS.
And then it could be connected to VRCFaceTracking and use this data for VRChat!

[zmerp]

ShinyQuagsire started working on it and then abandoned the idea because he couldn't make it work in any usable form. He's the most expert on this part so i'm pretty sure there is nothing we can do.

[blacktheon]

oh no... Is there any chance that we ask for the persona right and stream the video into the PC.
And then users could use some 3rd party software to extract the face tracking data?
I have tried some other software that supports live streaming persona video into PC, but they are not able to run in parallel with ALVR.

[zmerp]

This seems impractical. @shinyquagsire23 may know more

[shinyquagsire23]

yeah so the issue basically boils down to:

• There's no ARKit API for face tracking
• The virtual Persona webcam is buggy and restrictive: It has to be attached to a window, and you don't get to choose which one it attaches to. RealityKit spaces count as a window, but their position is a weird 0,0,0 location where you entered. So the camera will point at your back usually.
• If you turn away from the Persona window, it records the back of your head.
• visionOS doesn't allow repositioning windows for security reasons, so you can't force the Persona webcam to always follow your face

[blacktheon]

@zarik5 @shinyquagsire23 Thank you so much for your reply.
I am a Vision Pro VR experience developer as well. I found all these issues you mentioned are true.
I hope that there will be more and more APIs released for developers in the future.

[czyz]

I wonder if the developer of this app (link to reddit announcement of "Persona Studio") did anything extra-special to do Persona captures?

[Dog]

Based on the demos, it appears to have the same problem described above. The capture is done relative to the window placement, so if you face away from the capture window you get the back of the head.