Merge pull request #179 from justadudewhohacks/expressions

face expression recognition

Author: justadudewhohacks

README.md

@@ -16,6 +16,7 @@ Table of Contents:
   * **[Face Detection Models](#models-face-detection)**
   * **[68 Point Face Landmark Detection Models](#models-face-landmark-detection)**
   * **[Face Recognition Model](#models-face-recognition)**
+  * **[Face Expression Recognition Model](#models-face-expression-recognition)**
 * **[Getting Started](#getting-started)**
   * **[face-api.js for the Browser](#getting-started-browser)**
   * **[face-api.js for Nodejs](#getting-started-nodejs)**
@@ -61,6 +62,10 @@ Check out my face-api.js tutorials:
 
 ![preview_face-similarity](https://user-images.githubusercontent.com/31125521/40316573-0a1190c0-5d1f-11e8-8797-f6deaa344523.gif)
 
+## Face Expression Recognition
+
+![preview_face-expression-recognition](https://user-images.githubusercontent.com/31125521/50575270-f501d080-0dfb-11e9-9676-8f419efdade4.png)
+
 ## Face Landmark Detection
 
 ![face_landmarks_boxes_2](https://user-images.githubusercontent.com/31125521/46063404-00928b00-c16d-11e8-8f29-e9c50afd2bc8.jpg)
@@ -161,6 +166,12 @@ The neural net is equivalent to the **FaceRecognizerNet** used in [face-recognit
 
 The size of the quantized model is roughly 6.2 MB (**face_recognition_model**).
 
+<a name="models-face-expression-recognition"></a>
+
+## Face Expression Recognition Model
+
+The face expression recognition model is lightweight, fast and provides reasonable accuracy. The model has a size of roughly 310kb and it employs depthwise separable convolutions and densely connected blocks. It has been trained on a variety of images from publicly available datasets as well as images scraped from the web. Note, that wearing glasses might decrease the accuracy of the prediction results.
+
 <a name="getting-started"></a>
 
 # Getting Started
@@ -228,6 +239,7 @@ await faceapi.loadSsdMobilenetv1Model('/models')
 // await faceapi.loadFaceLandmarkModel('/models')
 // await faceapi.loadFaceLandmarkTinyModel('/models')
 // await faceapi.loadFaceRecognitionModel('/models')
+// await faceapi.loadFaceExpressionModel('/models')
 ```
 
 All global neural network instances are exported via faceapi.nets:
@@ -319,13 +331,13 @@ You can tune the options of each face detector as shown [here](#usage-face-detec
 
 **After face detection, we can furthermore predict the facial landmarks for each detected face as follows:**
 
-Detect all faces in an image + computes 68 Point Face Landmarks for each detected face. Returns **Array<[FaceDetectionWithLandmarks](#interface-face-detection-with-landmarks)>**:
+Detect all faces in an image + computes 68 Point Face Landmarks for each detected face. Returns **Array<[WithFaceLandmarks<WithFaceDetection<{}>>](#usage-utility-classes)>**:
 
 ``` javascript
 const detectionsWithLandmarks = await faceapi.detectAllFaces(input).withFaceLandmarks()
 ```
 
-Detect the face with the highest confidence score in an image + computes 68 Point Face Landmarks for that face. Returns **[FaceDetectionWithLandmarks](#interface-face-detection-with-landmarks) | undefined**:
+Detect the face with the highest confidence score in an image + computes 68 Point Face Landmarks for that face. Returns **[WithFaceLandmarks<WithFaceDetection<{}>>](#usage-utility-classes) | undefined**:
 
 ``` javascript
 const detectionWithLandmarks = await faceapi.detectSingleFace(input).withFaceLandmarks()
@@ -342,16 +354,52 @@ const detectionsWithLandmarks = await faceapi.detectAllFaces(input).withFaceLand
 
 **After face detection and facial landmark prediction the face descriptors for each face can be computed as follows:**
 
-Detect all faces in an image + computes 68 Point Face Landmarks for each detected face. Returns **Array<[FullFaceDescription](#interface-full-face-description)>**:
+Detect all faces in an image + computes 68 Point Face Landmarks for each detected face. Returns **Array<[WithFaceDescriptor<WithFaceLandmarks<WithFaceDetection<{}>>>](#usage-utility-classes)>**:
+
+``` javascript
+const results = await faceapi.detectAllFaces(input).withFaceLandmarks().withFaceDescriptors()
+```
+
+Detect the face with the highest confidence score in an image + computes 68 Point Face Landmarks and face descriptor for that face. Returns **[WithFaceDescriptor<WithFaceLandmarks<WithFaceDetection<{}>>>](#usage-utility-classes) | undefined**:
+
+``` javascript
+const result = await faceapi.detectSingleFace(input).withFaceLandmarks().withFaceDescriptor()
+```
+
+### Recognizing Face Expressions
+
+**Face expressions recognition can be performed for detected faces as follows:**
+
+Detect all faces in an image + recognize face expressions. Returns **Array<[WithFaceExpressions<WithFaceDetection<{}>>](#usage-utility-classes)>**:
+
+``` javascript
+const detectionsWithExpressions = await faceapi.detectAllFaces(input).withFaceExpressions()
+```
+
+Detect the face with the highest confidence score in an image + recognize the face expression for that face. Returns **[WithFaceExpressions<WithFaceDetection<{}>>](#usage-utility-classes) | undefined**:
 
 ``` javascript
-const fullFaceDescriptions = await faceapi.detectAllFaces(input).withFaceLandmarks().withFaceDescriptors()
+const detectionWithExpressions = await faceapi.detectSingleFace(input).withFaceExpressions()
 ```
 
-Detect the face with the highest confidence score in an image + computes 68 Point Face Landmarks and face descriptor for that face. Returns **[FullFaceDescription](#interface-full-face-description) | undefined**:
+### Composition of Tasks
+
+**Tasks can be composed as follows:**
 
 ``` javascript
-const fullFaceDescription = await faceapi.detectSingleFace(input).withFaceLandmarks().withFaceDescriptor()
+// all faces
+await faceapi.detectAllFaces(input)
+await faceapi.detectAllFaces(input).withFaceExpressions()
+await faceapi.detectAllFaces(input).withFaceLandmarks()
+await faceapi.detectAllFaces(input).withFaceExpressions().withFaceLandmarks()
+await faceapi.detectAllFaces(input).withFaceExpressions().withFaceLandmarks().withFaceDescriptors()
+
+// single face
+await faceapi.detectSingleFace(input)
+await faceapi.detectSingleFace(input).withFaceExpressions()
+await faceapi.detectSingleFace(input).withFaceLandmarks()
+await faceapi.detectSingleFace(input).withFaceExpressions().withFaceLandmarks()
+await faceapi.detectSingleFace(input).withFaceExpressions().withFaceLandmarks().withFaceDescriptor()
 ```
 
 ### Face Recognition by Matching Descriptors
@@ -361,43 +409,43 @@ To perform face recognition, one can use faceapi.FaceMatcher to compare referenc
 First, we initialize the FaceMatcher with the reference data, for example we can simply detect faces in a **referenceImage** and match the descriptors of the detected faces to faces of subsquent images:
 
 ``` javascript
-const fullFaceDescriptions = await faceapi
+const results = await faceapi
   .detectAllFaces(referenceImage)
   .withFaceLandmarks()
   .withFaceDescriptors()
 
-if (!fullFaceDescriptions.length) {
+if (!results.length) {
   return
 }
 
 // create FaceMatcher with automatically assigned labels
 // from the detection results for the reference image
-const faceMatcher = new faceapi.FaceMatcher(fullFaceDescriptions)
+const faceMatcher = new faceapi.FaceMatcher(results)
 ```
 
 Now we can recognize a persons face shown in **queryImage1**:
 
 ``` javascript
-const singleFullFaceDescription = await faceapi
+const singleResult = await faceapi
   .detectSingleFace(queryImage1)
   .withFaceLandmarks()
   .withFaceDescriptor()
 
-if (singleFullFaceDescription) {
-  const bestMatch = faceMatcher.findBestMatch(singleFullFaceDescription.descriptor)
+if (singleResult) {
+  const bestMatch = faceMatcher.findBestMatch(singleResult.descriptor)
   console.log(bestMatch.toString())
 }
 ```
 
 Or we can recognize all faces shown in **queryImage2**:
 
 ``` javascript
-const fullFaceDescriptions = await faceapi
+const results = await faceapi
   .detectAllFaces(queryImage2)
   .withFaceLandmarks()
   .withFaceDescriptors()
 
-fullFaceDescriptions.forEach(fd => {
+results.forEach(fd => {
   const bestMatch = faceMatcher.findBestMatch(fd.descriptor)
   console.log(bestMatch.toString())
 })
@@ -430,7 +478,7 @@ Drawing the detected faces into a canvas:
 const detections = await faceapi.detectAllFaces(input)
 
 // resize the detected boxes in case your displayed image has a different size then the original
-const detectionsForSize = detections.map(det => det.forSize(input.width, input.height))
+const detectionsForSize = faceapi.resizeResults(detections, { width: input.width, height: input.height })
 // draw them into a canvas
 const canvas = document.getElementById('overlay')
 canvas.width = input.width
@@ -446,7 +494,7 @@ const detectionsWithLandmarks = await faceapi
   .withFaceLandmarks()
 
 // resize the detected boxes and landmarks in case your displayed image has a different size then the original
-const detectionsWithLandmarksForSize = detectionsWithLandmarks.map(det => det.forSize(input.width, input.height))
+const detectionsWithLandmarksForSize = faceapi.resizeResults(detectionsWithLandmarks, { width: input.width, height: input.height })
 // draw them into a canvas
 const canvas = document.getElementById('overlay')
 canvas.width = input.width
@@ -579,27 +627,55 @@ export interface IFaceLandmarks {
 }
 ```
 
-<a name="interface-face-detection-with-landmarks"></a>
+<a name="with-face-detection"></a>
 
-### IFaceDetectionWithLandmarks
+### WithFaceDetection
 
 ``` javascript
-export interface IFaceDetectionWithLandmarks {
+export type WithFaceDetection<TSource> TSource & {
   detection: FaceDetection
+}
+```
+
+<a name="with-face-landmarks"></a>
+
+### WithFaceLandmarks
+
+``` javascript
+export type WithFaceLandmarks<TSource> TSource & {
+  unshiftedLandmarks: FaceLandmarks
   landmarks: FaceLandmarks
+  alignedRect: FaceDetection
 }
 ```
 
-<a name="interface-full-face-description"></a>
+<a name="with-face-descriptor"></a>
 
-### IFullFaceDescription
+### WithFaceDescriptor
 
 ``` javascript
-export interface IFullFaceDescription extends IFaceDetectionWithLandmarks {
+export type WithFaceDescriptor<TSource> TSource & {
   descriptor: Float32Array
 }
 ```
 
+<a name="with-face-expressions"></a>
+
+### WithFaceExpressions
+
+``` javascript
+export type FaceExpression = 'neutral' | 'happy' | 'sad' | 'angry' | 'fearful' | 'disgusted' | 'surprised'
+
+export type FaceExpressionPrediction = {
+  expression: FaceExpression,
+  probability: number
+}
+
+export type WithFaceExpressions<TSource> TSource & {
+  expressions: FaceExpressionPrediction[]
+}
+```
+
 <a name="other-useful-utility"></a>
 
 ## Other Useful Utility

examples/examples-browser/public/js/commons.js

@@ -25,8 +25,16 @@ async function requestExternalImage(imageUrl) {
 function renderNavBar(navbarId, exampleUri) {
   const examples = [
     {
-      uri: 'face_and_landmark_detection',
-      name: 'Face And Landmark Detection'
+      uri: 'face_detection',
+      name: 'Face Detection'
+    },
+    {
+      uri: 'face_landmark_detection',
+      name: 'Face Landmark Detection'
+    },
+    {
+      uri: 'face_expression_recognition',
+      name: 'Face Expression Recognition'
     },
     {
       uri: 'face_recognition',
@@ -41,8 +49,16 @@ function renderNavBar(navbarId, exampleUri) {
       name: 'Video Face Tracking'
     },
     {
-      uri: 'webcam_face_tracking',
-      name: 'Webcam Face Tracking'
+      uri: 'webcam_face_detection',
+      name: 'Webcam Face Detection'
+    },
+    {
+      uri: 'webcam_face_landmark_detection',
+      name: 'Webcam Face Landmark Detection'
+    },
+    {
+      uri: 'webcam_face_expression_recognition',
+      name: 'Webcam Face Expression Recognition'
     },
     {
       uri: 'bbt_face_landmark_detection',
@@ -112,7 +128,7 @@ function renderNavBar(navbarId, exampleUri) {
         li.style.background='#b0b0b0'
       }
       const a = document.createElement('a')
-      a.classList.add('waves-effect', 'waves-light')
+      a.classList.add('waves-effect', 'waves-light', 'pad-sides-sm')
       a.href = ex.uri
       const span = document.createElement('span')
       span.innerHTML = ex.name
@@ -123,7 +139,7 @@ function renderNavBar(navbarId, exampleUri) {
     })
 
   $('.button-collapse').sideNav({
-    menuWidth: 240
+    menuWidth: 260
   })
 }
 

examples/examples-browser/public/js/drawing.js

@@ -7,7 +7,7 @@ function resizeCanvasAndResults(dimensions, canvas, results) {
 
   // resize detections (and landmarks) in case displayed image is smaller than
   // original size
-  return results.map(res => res.forSize(width, height))
+  return faceapi.resizeResults(results, { width, height })
 }
 
 function drawDetections(dimensions, canvas, detections) {
@@ -29,4 +29,14 @@ function drawLandmarks(dimensions, canvas, results, withBoxes = true) {
     color: 'green'
   }
   faceapi.drawLandmarks(canvas, faceLandmarks, drawLandmarksOptions)
+}
+
+function drawExpressions(dimensions, canvas, results, thresh, withBoxes = true) {
+  const resizedResults = resizeCanvasAndResults(dimensions, canvas, results)
+
+  if (withBoxes) {
+    faceapi.drawDetection(canvas, resizedResults.map(det => det.detection), { withScore: false })
+  }
+
+  faceapi.drawFaceExpressions(canvas, resizedResults.map(({ detection, expressions }) => ({ position: detection.box, expressions })))
 }

examples/examples-browser/public/js/imageSelectionControls.js

@@ -10,8 +10,21 @@ async function loadImageFromUrl(url) {
   updateResults()
 }
 
-function renderImageSelectList(selectListId, onChange, initialValue) {
-  const images = [1, 2, 3, 4, 5].map(idx => `bbt${idx}.jpg`)
+function renderImageSelectList(selectListId, onChange, initialValue, withFaceExpressionImages) {
+  let images = [1, 2, 3, 4, 5].map(idx => `bbt${idx}.jpg`)
+
+  if (withFaceExpressionImages) {
+    images = [
+      'happy.jpg',
+      'sad.jpg',
+      'angry.jpg',
+      'disgusted.jpg',
+      'surprised.jpg',
+      'fearful.jpg',
+      'neutral.jpg'
+    ].concat(images)
+  }
+
   function renderChildren(select) {
     images.forEach(imageName =>
       renderOption(
@@ -30,13 +43,14 @@ function renderImageSelectList(selectListId, onChange, initialValue) {
   )
 }
 
-function initImageSelectionControls() {
+function initImageSelectionControls(initialValue = 'bbt1.jpg', withFaceExpressionImages = false) {
   renderImageSelectList(
     '#selectList',
     async (uri) => {
       await onSelectedImageChanged(uri)
     },
-    'bbt1.jpg'
+    initialValue,
+    withFaceExpressionImages
   )
   onSelectedImageChanged($('#selectList select').val())
 }

examples/examples-browser/public/styles.css

@@ -3,7 +3,7 @@
   right: 0;
   margin: auto;
   margin-top: 20px;
-  padding-left: 260px;
+  padding-left: 280px;
   display: inline-flex !important;
 }
 
@@ -53,6 +53,10 @@
   padding: 0 10px !important;
 }
 
+.pad-sides-sm {
+  padding: 0 8px !important;
+}
+
 #github-link {
   display: flex !important;
   justify-content: center;