iPad Mouth Sync - Animated Avatar App

A creative iPad app built with Expo that lets users draw mouth positions for animated characters, generate AI characters, and have interactive conversations with voice synthesis.

Features

• Mouth Drawing System: Draw 12 different mouth positions for realistic lip-sync animation
• AI Character Generation: Create unique characters using AI (OpenAI DALL-E integration)
• Interactive Positioning: Drag, resize, and rotate mouth placement on characters
• Voice Synthesis: Text-to-speech with synchronized mouth animations
• Chat Interface: Have conversations with your animated avatar
• Multiple Themes: Dark, Light, Neon, and Pastel color schemes
• Local Storage: Save all mouth drawings and characters on device
• Touch Optimized: Designed specifically for iPad with gesture support

Setup

1. Install dependencies:

npm install

2. Configure API Key:

   • Open src/services/ai.ts
   • Replace YOUR_OPENAI_API_KEY with your actual OpenAI API key

3. Run the app:

# For iOS (iPad)
npm run ios

# For development
npx expo start

How to Use

Step 1: Draw Mouth Positions

• Draw or upload images for 12 different mouth shapes
• Each position corresponds to specific sounds (phonemes)
• Use different colors and brush sizes
• All drawings are saved automatically

Step 2: Generate Character

• Describe your character or leave blank for random generation
• AI will create a character without a mouth
• Characters are saved to your device

Step 3: Position the Mouth

• Drag to position the mouth on your character
• Use sliders to adjust size and rotation
• Find the perfect placement for natural animation

Step 4: Chat & Animate

• Type messages to chat with your avatar
• Avatar responds with AI-generated text
• Mouth animates in sync with speech
• Voice synthesis brings your character to life

Technical Stack

• Framework: React Native with Expo
• Drawing: @shopify/react-native-skia
• Storage: AsyncStorage
• Voice: expo-speech
• Animations: react-native-reanimated
• AI Integration: OpenAI API (GPT-4 & DALL-E)

Project Structure

expo-ipad-mouth-sync/
├── src/
│   ├── components/
│   │   ├── DrawingCanvas.tsx      # Mouth drawing interface
│   │   ├── MouthPositioner.tsx    # Positioning interface
│   │   └── AnimatedAvatar.tsx     # Avatar with animations
│   ├── screens/
│   │   ├── DrawMouthsScreen.tsx   # Mouth drawing workflow
│   │   └── ChatScreen.tsx         # Chat interface
│   ├── services/
│   │   ├── ai.ts                  # AI integration
│   │   └── storage.ts             # Local storage
│   ├── constants/
│   │   └── mouthPositions.ts      # Phoneme definitions
│   └── types/
│       └── index.ts               # TypeScript definitions
└── App.tsx                        # Main app component

Mouth Positions

The app uses 12 standard mouth positions for animation:

• Neutral (rest position)
• Ah (open)
• Ee (smile)
• Oh (round)
• Oo (pucker)
• Mm (closed)
• F/V
• Th
• Ch/Sh
• L
• R
• W

Customization

Themes

Access theme selector with the 🎨 button to choose from:

• Dark (default)
• Light
• Neon
• Pastel

API Configuration

Update src/services/ai.ts to use your preferred AI service or model.

Requirements

• iOS 13.0+ / iPad
• Node.js 16+
• Expo SDK 49+
• OpenAI API key (for character generation and chat)

License

MIT