Xverse VRFace Tracking is a lower face tracking solution developed by XVERSE Technology Inc. (Shenzhen, China), aiming to provide real-time face tracking with any VR headset in different lighting. We have trained a new deep learning model to better detect facial expression changes. Currently, our face tracking project is implemented based on VRface and VRCFaceTracking，leveraging the simplicity and convenience of VRCFT.
We will actively release new features in this repo, please stay tuned. Some future updates will contain:
- Enhance Facial Expression Prediction
- Inplement Tongue Expression Prediction
- Extend Hardware Support for VR Devices
Below are examples of our method applied to different VRFace avatars.
You can make your own camera hardware following Hardware or contact us via email xengine@xverse.cn to purchase finished products. Below are visuals of the camera’s appearance, installation, and how it looks in use.
- Windows 10 or 11
- Python 3.12
- VRface and VRCFaceTracking
The file structure of this project is as below:
├── software/
│ ├── CameraIPReciver.py # camera IP reciver
│ └── XverseVRfaceMouthDetectionUI.py # lower face detection UI
├── CameraWebServer/ # Camera web service
│ ├── app_httpd.cpp # HTTP server implementation
│ ├── CameraWebServer.ino # Arduino main sketch
│ ├── camera_index.h # Web interface template
│ ├── camera_pins.h # Hardware pin definitions
│ ├── ci.json # CI configuration
│ ├── partitions.csv # ESP32 partition table
│ ├── README.md # README doc
│ ├── remote_post.cpp # Remote communication module
│ └── remote_post.h # Communication header
├── assets/ # plugin folder
│ ├── VRCFaceTracking.Xverse.dll # VRCFT plugin
│ └── XverseConfig.json # plugin json
├── images/ # pictures for readme
│ └── ...
├── models/ # onnx model
│ └── XVRFaceTracking.onnx # onnx model for lower face tracking
├── 3D_hardware/ # hardware files for 3D print
│ ├── hardware-led.blend # blend file for 3D print
│ └── hardware-led.stl # stl file for 3D print
├── requirements.txt # Python requriements
├── README.md
├── LICENSE
0.Install the dependency packages listed in requriements.txt.
pip install -r requirements.txt
1.Install ardunio. Connect ESP32S3 to your PC with a micro-USB/mini-USB/USB type-c cable. Open CameraWebServer.ino in CameraWebServer via ardunio. Change board type to XIAO_ESP32S3 and fill your WiFi network name and password.
Open remote_post.cpp in CameraWebServer. Fill in your PC IP, verify and upload.
You can run 'ipconfig' in terminal to know your PC IP.
After successful upload, the arduino will display the notification shown in the figure below.
After uploading CameraWebServer to ESP32S3, run CameraIPReciver.py to recieve Stream URL. Ensure that ports 8889 on your PC is not blocked or already in use.
2.Install VRCFaceTracking. Drop the VRCFaceTracking.Xverse.dll and XverseConfig.json into AppData\Roaming\VRCFaceTracking\CustomLibs. If you can't find this path, you can use Everything for search. If this folder does not exist you can create it, VRCFaceTracking will create it on launch.
3.Run XverseVRfaceMouthDetectionUI.py. Stream Url is from CameraIPReciver.py(step 1). Remeber to keep 'http://' in Stream Url. ONNX file is our pretrained deep learning model in folder model. ONNX Path refers to the file path of the ONNX model on your computer. Generally, the default ONNX path should work. ROI settings can change the region of your capture picture. For OneEuro filter, you can check the official website for reference.
4.Open VRCFaceTracking and VRChat. In VRChat, select an avatar that supports VRCFT and enable OCT. If you are unable to animate the avatar's facial expressions, ensure that ports 8888 and 9000 on your local machine are not blocked or already in use. If you are unable to establish an OSC connection between your PC and VR, please refer to VRCFT website. Remember to open OSC in your VR device.
You will need Xiao_ESP32S3(It has compact size with both wireless and wired support and no need for additional antennas) and OV2640 with a viewing angle of 160 degrees. These two components, when powered through the Type-C port, can stream the signal to the computer. Alternative device for Xiao_ESP32S3 is Freenove ESP 32-S3 WROOM, which is larger and more expensive than Xiao_ESP32S3, and lacks support for an external antenna. Alternative cameras with a narrower viewing angle may reduce recognition accuracy and demand higher lighting conditions.
Optional components: 3.7V battery, 5mm White Through Hole LED. Either 3.3V or 5V can be used to power the LEDs, however the resistors used will be different. 5V can only be used when powered over USB while 3.3V can be used with both USB and battery power. For 3.3V power: Use 82-ohm resistors in series with the LEDs. For 5V power: Use 160-ohm resistors in series with the LEDs. Connect the GND pin on the ESP32 to the cathode (short leg) of the LED. Add an appropriate resistor (82 ohms for 3.3V, 160 ohms for 5V) in series with the anode (long leg) of the LED. Do not connect LEDs directly to the ESP32 pins without a resistor, as it may damage the LEDs or the microcontroller. Refer to the following figure for LED connection.
If you want to connect a battery to the Xiao, we recommend that you purchase a qualified 3.7V rechargeable lithium battery. When soldering the battery, be careful to distinguish the positive and negative terminals. The negative terminal of the battery should be on the side closest to the USB port (BAT-), and the positive terminal of the battery should be on the side away from the USB port (BAT+). Refer to the following two figures for battery connection.
Print the model inside folder '3D_hardware'. Currently, our 3D model is provided for the Quest 3. If you require hardware-related assistance or wish to purchase finished products, please contact us via email xengine@xverse.cn.