Automatic interpretation of affective facial expressions in the context of interpersonal interaction

Research output: Contribution to journalArticleAcademicpeer-review

17 Citations (Scopus)
93 Downloads (Pure)

Abstract

This paper proposes a method for interpretation of the emotions detected in facial expressions in the context of the events that cause them. The method was developed to analyze the video recordings of facial expressions depicted during a collaborative game played as a part of the Mars-500 experiment. In this experiment, six astronauts were isolated for 520 days in a space station to simulate a flight to Mars. Seven time-dependent components of facial expressions were extracted from the video recordings of the experiment. To interpret these dynamic components, we proposed a mathematical model of emotional events. Genetic programming was used to find the locations, types, and intensities of the emotional events as well as the way the recorded facial expressions represented reactions to them. By classification of different statistical properties of the data, we found that there are significant relations between the facial expressions of different crew members and a memory effect between the collective emotional states of the crew members. The model of emotional events was validated on previously unseen video recordings of the astronauts. We demonstrated that both genetic search and optimization of the parameters improve the accuracy of the proposed model. This method is a step toward automating the analysis of affective expressions in terms of the cognitive appraisal theory of emotion, which relies on the dependence of the expressed emotion on the causing event.

Original languageEnglish
Article number7105880
Pages (from-to)409-418
Number of pages10
JournalIEEE Transactions on Human-Machine Systems
Volume45
Issue number4
DOIs
Publication statusPublished - 1 Aug 2015

Fingerprint

Video recording
facial expression
video recording
interpretation
event
interaction
emotion
Genetic programming
Experiments
Space stations
experiment
Mathematical models
Data storage equipment
flight
programming
cause

Keywords

  • Appraisal theory of emotion
  • interpretation of affective expressions
  • Mars-500
  • mathematical modeling of affective interactions
  • second-person perspective

Cite this

@article{e387d669910943dd88978375dd63efb9,
title = "Automatic interpretation of affective facial expressions in the context of interpersonal interaction",
abstract = "This paper proposes a method for interpretation of the emotions detected in facial expressions in the context of the events that cause them. The method was developed to analyze the video recordings of facial expressions depicted during a collaborative game played as a part of the Mars-500 experiment. In this experiment, six astronauts were isolated for 520 days in a space station to simulate a flight to Mars. Seven time-dependent components of facial expressions were extracted from the video recordings of the experiment. To interpret these dynamic components, we proposed a mathematical model of emotional events. Genetic programming was used to find the locations, types, and intensities of the emotional events as well as the way the recorded facial expressions represented reactions to them. By classification of different statistical properties of the data, we found that there are significant relations between the facial expressions of different crew members and a memory effect between the collective emotional states of the crew members. The model of emotional events was validated on previously unseen video recordings of the astronauts. We demonstrated that both genetic search and optimization of the parameters improve the accuracy of the proposed model. This method is a step toward automating the analysis of affective expressions in terms of the cognitive appraisal theory of emotion, which relies on the dependence of the expressed emotion on the causing event.",
keywords = "Appraisal theory of emotion, interpretation of affective expressions, Mars-500, mathematical modeling of affective interactions, second-person perspective",
author = "E.I. Barakova and R. Gorbunov and G.W.M. Rauterberg",
year = "2015",
month = "8",
day = "1",
doi = "10.1109/THMS.2015.2419259",
language = "English",
volume = "45",
pages = "409--418",
journal = "IEEE Transactions on Human-Machine Systems",
issn = "2168-2291",
publisher = "IEEE Systems, Man, and Cybernetics Society",
number = "4",

}

Automatic interpretation of affective facial expressions in the context of interpersonal interaction. / Barakova, E.I.; Gorbunov, R.; Rauterberg, G.W.M.

In: IEEE Transactions on Human-Machine Systems, Vol. 45, No. 4, 7105880, 01.08.2015, p. 409-418.

Research output: Contribution to journalArticleAcademicpeer-review

TY - JOUR

T1 - Automatic interpretation of affective facial expressions in the context of interpersonal interaction

AU - Barakova, E.I.

AU - Gorbunov, R.

AU - Rauterberg, G.W.M.

PY - 2015/8/1

Y1 - 2015/8/1

N2 - This paper proposes a method for interpretation of the emotions detected in facial expressions in the context of the events that cause them. The method was developed to analyze the video recordings of facial expressions depicted during a collaborative game played as a part of the Mars-500 experiment. In this experiment, six astronauts were isolated for 520 days in a space station to simulate a flight to Mars. Seven time-dependent components of facial expressions were extracted from the video recordings of the experiment. To interpret these dynamic components, we proposed a mathematical model of emotional events. Genetic programming was used to find the locations, types, and intensities of the emotional events as well as the way the recorded facial expressions represented reactions to them. By classification of different statistical properties of the data, we found that there are significant relations between the facial expressions of different crew members and a memory effect between the collective emotional states of the crew members. The model of emotional events was validated on previously unseen video recordings of the astronauts. We demonstrated that both genetic search and optimization of the parameters improve the accuracy of the proposed model. This method is a step toward automating the analysis of affective expressions in terms of the cognitive appraisal theory of emotion, which relies on the dependence of the expressed emotion on the causing event.

AB - This paper proposes a method for interpretation of the emotions detected in facial expressions in the context of the events that cause them. The method was developed to analyze the video recordings of facial expressions depicted during a collaborative game played as a part of the Mars-500 experiment. In this experiment, six astronauts were isolated for 520 days in a space station to simulate a flight to Mars. Seven time-dependent components of facial expressions were extracted from the video recordings of the experiment. To interpret these dynamic components, we proposed a mathematical model of emotional events. Genetic programming was used to find the locations, types, and intensities of the emotional events as well as the way the recorded facial expressions represented reactions to them. By classification of different statistical properties of the data, we found that there are significant relations between the facial expressions of different crew members and a memory effect between the collective emotional states of the crew members. The model of emotional events was validated on previously unseen video recordings of the astronauts. We demonstrated that both genetic search and optimization of the parameters improve the accuracy of the proposed model. This method is a step toward automating the analysis of affective expressions in terms of the cognitive appraisal theory of emotion, which relies on the dependence of the expressed emotion on the causing event.

KW - Appraisal theory of emotion

KW - interpretation of affective expressions

KW - Mars-500

KW - mathematical modeling of affective interactions

KW - second-person perspective

UR - http://www.scopus.com/inward/record.url?scp=84937570845&partnerID=8YFLogxK

U2 - 10.1109/THMS.2015.2419259

DO - 10.1109/THMS.2015.2419259

M3 - Article

VL - 45

SP - 409

EP - 418

JO - IEEE Transactions on Human-Machine Systems

JF - IEEE Transactions on Human-Machine Systems

SN - 2168-2291

IS - 4

M1 - 7105880

ER -