EmoTune is a real-time, single-user, emotion-responsive system that combines facial and vocal emotion analysis with adaptive music generation. It explores AI-driven music therapy for mental well-being by dynamically responding to users' affective states.
EmoTune captures and analyzes emotional signals through facial expressions and vocal tone, interprets the user's current affective state (valence, arousal, and discrete emotion), and adapts music output in real time. The system is modular, with independent perception, feedback, and synthesis components.
- Multimodal Emotion Analysis: Real-time facial and vocal emotion recognition using deep learning (EmoNet, MTCNN, etc.).
- Valence-Arousal Mapping: Continuous affective state estimation and trajectory tracking.
- Adaptive Music Generation: Dynamic music parameterization and synthesis via FoxDot, Sonic Pi, MIDI, or TidalCycles engines.
- Reinforcement Learning: Soft Actor-Critic (SAC) agent adapts music parameters to optimize user affect and feedback.
- User Feedback Integration: Collects explicit (ratings, comfort, effectiveness, comments) and implicit feedback for RL.
- Web Dashboard: Interactive frontend for session control, emotion visualization, trajectory tracking, and feedback.
- Session Logging: Stores emotion, music, and feedback data for analysis and research.
- Backend: Python (Flask, Flask-SocketIO), modular core (emotion, music, trajectory, feedback, RL), threaded session manager.
- Frontend: HTML/JS dashboard
- RL Policy: Soft Actor-Critic (SAC) with a two-layer neural network policy
- Emotion Models: Uses vendor/emonet for facial emotion, MTCNN for face detection, and Wav2Vec2 for voice.
- Clone the repository (with submodules):
git clone --recurse-submodules <repo-url> cd emotune
- Install dependencies (Python 3.9+ recommended):
pip install -r requirements.txt - Download/prep emotion model weights (see
vendor/emonet/pretrained/).
- Start the server:
python -m emotune - Open the dashboard:
- Visit http://localhost:5000 in your browser.
- Start a session:
- Select a trajectory and duration, then click "Start Session".
- Allow camera/microphone access for emotion capture.
- View real-time emotion, music, and trajectory feedback.
- Submit feedback at session end.
- Edit
config/default.jsonfor system/session parameters. - Music and trajectory templates:
emotune/config/music_params.py,emotune/config/trajectories.py.
emotune/core/— Main logic (emotion, music, RL, feedback, trajectory)emotune/web/— Flask app, dashboard, static assetsvendor/emonet/— Facial emotion model (submodule)logs/— Session logs (emotion, music, feedback, performance)tests/— Example tests
- Modular for research: swap models, add new feedback types, extend RL, or integrate new music engines.
- Feedback and emotion logs support offline analysis.
- RL reward and state logic can be customized for new experiments.
- Python 3.9+
- Camera and microphone for emotion capture
- EmoNet for facial emotion recognition
- FoxDot, Sonic Pi, TidalCycles for music engines