A context-aware design for emotion recognition in natural settings -

Abstract

A major issue in achieving automated emotion recognition systems that perform well outside the lab is the use of real-world data that reflects the occurrence of emotions in everyday life. Furthermore, situational context contains emotion-relevant information that should be included in any multimodal system for emotion recognition out of the lab. The majority of studies in machine emotion recognition have been based on restricted laboratory environments where data is collected by inducing emotional responses through experimental design rather than observing natural emotions in everyday life. Moreover, models typically rely on classical modalities such as physiological response, audio, and facial expressions, while ignoring the information inherent in the user's situational environment, even though it has been shown that human perception of emotions occurs in context. In this thesis, a design is proposed for an emotion recognition model that combines physiological data with context data from the real-world. A user study is conducted to collect real-world emotion and context data from participants using a mobile application. The performance of different classification models is compared for the task of recognizing emotions on the Valence-Arousal scale. It is shown that including context with the Bayesian Network model and the K-Nearest-Neighbors models improves the performance of the physiological model particularly by increasing the recall and F-score for minority classes. In fact, context alone as a separate classifier is shown to outperform the physiological classifier in several cases. Models customized to participants increased performance by increasing the effect of context. Finally, an analysis of the information gain of the context features showed that the context features which contained the most emotion-relevant information were the relationship and presence of nearby people to the users, as well as users’ current activity.