Hierarchical Classification and Transfer Learning to Recognize Head Gestures and Facial Expressions Using Earbuds

Head gestures and facial expressions - like, e.g., nodding or smiling - are important indicators of the quality of human interactions in physical meetings as well as in computer-mediated settings. Computer systems able to recognize such behavioral cues can support and improve human interactions. Several researchers have thus tackled the problem of automatically recognizing head gestures and facial expressions, mainly leveraging video data. In this paper, we instead consider inertial signals collected from unobtrusive, ear-mounted devices. We focus on typical activities performed during social interactions - head shaking, nodding, smiling, talking and yawning - and propose a hierarchical classification approach to discriminate them from each other. Further, we investigate whether the transfer of knowledge learned from publicly available datasets leads to further performance improvements. Our results show that the combined use of our hierarchical approach and transfer learning allows the classifier to discriminate head and mouth activities with an F1 score of 84.79, smile, talk and yawn with an F1 score of 45.42, and nodding and head shaking with an F1 score of 88.24, outperforming shallow classifiers by 2-9 percentage points.