Facemoji Kit

Bring Facemoji into your app and let users express themselves more comfortably—or their alter ego—in fun, new ways with privacy. Unlike lenses or AR filters that are one-size-fits-all, Facemoji avatars can be personalized.

Facemoji Kit provides an accurate 3D head pose in world space, 51 facial expressions, and optionally a dense 3D mesh of a person’s face. Facemoji Kit uses blend shapes/morph targets (FACS and ARKit-compatible) to animate a 2D or 3D character in ways that follow the user’s facial expressions in real-time.

How it works

RGB Input → Magic NN ✨ → Blend shapes weights and pose → Avatar modelling and rendering

Key features

• Scalable facial tracking
  • A configurable trade-off between accuracy and speed
    • e.g. low-fidelity 2D vs. high-fidelity 3D avatars
• 1-2MB ML model size
• 3D reprojection to the input photo/video
• No avatar mode similar to Snap's Lenses or Instagram's AR filters
• Supported platforms
  • Unified iOS/Mac OS X Kit with internal platform optimizations
  • Platform suited API and packaging
  • Basic Windows and Android support, full Windows and Android Kits coming soon

Input

RGB image or video

Output

• FACS and ARKit-compatible blendshapes
• Gaze tracking
• Head position in 2D and 3D
• Head rotation and scale in world coordinates
• On desktop or high-end phones:
  • Dense mesh (up to 50 000 keypoints)
  • Face segmentation (per pixel)
  • UV and depth estimation

Tracking speed

Higher accuracy

• 60 FPS on iPhone SE
• 90 FPS on iPhone X or newer

Lower accuracy

• 90 FPS on iPhone SE
• 120 FPS on iPhone X or newer

Examples

• Facemoji, our flagship app using the Kit :)
• Gamebytes, games for iMessage
• Takeoff, live learning gameshows
• REALITY, avatar live streaming
• Airtime, video chat rooms

Demo and download

Get the TestFlight demo app or try one of the apps above. If you’re interested in licensing Facemoji Kit, have a question and want to chat, feel free to join our Discord.

3D Content

Facemoji Kit includes all the 3D accessories, models, textures, and shading assets to render Facemoji avatars in your app for your users. There are over 1,000 items, including hair styles, hats, sunglasses, face masks, animated accessories or even face tattoos. Accessories could be white-labeled.

Accessories

No avatar mode

Optional mode with no Facemoji and only 3D accessories with occluders.

Diversity