This goal of this project is to program Cozmo, a small, interactive robot developed by Anki, into a caring, emotionally intelligent robot companion and assistant using multimodal emotion recognition capabilities. The following tools and features are planned:
-
Cozmo Python SDK with built-in facial expression estimation
-
PyAudio for speech recording
-
Watson Developer Cloud Python SDK for speech to text service
-
GPT-4 for multi-turn conversational dialogue flow, text sentiment analysis, and enhancing conversational responses based on sentiment
-
Facial Emotion Recognition using Cozmo's [estimate_facial_expression feature] (https://developer.anki.com/vector/docs/generated/anki_vector.robot.html)
-
Real-time emotion recognizer based on acoustic properties of speech using the tool emoVoice
-
Bonus: Emotion recognition based on physiological signals processed from SSI (Social Signal Interpretation) framework
The target audience is intended to be all people who want a companion, entertainment, or a support system during difficult times.