Micro-expression detection in long videos using optical flow and recurrent neural networks

Michiel Verburg, Vlado Menkovski

Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

1 Citation (Scopus)
1 Downloads (Pure)

Abstract

Facial micro-expressions are subtle and involuntary expressions that can reveal concealed emotions. Microexpressions are an invaluable source of information in application domains such as lie detection, mental health, sentiment analysis and more. One of the biggest challenges in this field of research is the small amount of available spontaneous micro-expression data. However, spontaneous data collection is burdened by time-consuming and expensive annotation. Hence, methods are needed which can reduce the amount of data that annotators have to review. This paper presents a novel micro-expression spotting method using a recurrent neural network (RNN) on optical flow features. We extract Histogram of Oriented Optical Flow (HOOF) features to encode the temporal changes in selected face regions. Finally, the RNN spots short intervals which are likely to contain occurrences of relevant facial micro-movements. The proposed method is evaluated on the SAMM database. Any chance of subject bias is eliminated by training the RNN using Leave-One-Subject-Out cross-validation. Comparing the spotted intervals with the labeled data shows that the method produced 1569 false positives while obtaining a recall of 0.4654. The initial results show that the proposed method would reduce the video length by a factor of 3.5, while still retaining almost half of the relevant micro-movements. Lastly, as the model gets more data, it becomes better at detecting intervals, which the proposed method suitable for supporting the annotation process.

Original languageEnglish
Title of host publication14th IEEE International Conference on Automatic Face and Gesture Recognition, FG 2019
Place of PublicationPiscataway
PublisherInstitute of Electrical and Electronics Engineers
Number of pages6
ISBN (Electronic)978-1-7281-0089-0
DOIs
Publication statusPublished - 1 May 2019
Event14th IEEE International Conference on Automatic Face and Gesture Recognition, FG 2019 - Lille, France
Duration: 14 May 201918 May 2019

Conference

Conference14th IEEE International Conference on Automatic Face and Gesture Recognition, FG 2019
CountryFrance
CityLille
Period14/05/1918/05/19

Fingerprint

Optical flows
Recurrent neural networks
Health

Cite this

Verburg, M., & Menkovski, V. (2019). Micro-expression detection in long videos using optical flow and recurrent neural networks. In 14th IEEE International Conference on Automatic Face and Gesture Recognition, FG 2019 [8756588] Piscataway: Institute of Electrical and Electronics Engineers. https://doi.org/10.1109/FG.2019.8756588
Verburg, Michiel ; Menkovski, Vlado. / Micro-expression detection in long videos using optical flow and recurrent neural networks. 14th IEEE International Conference on Automatic Face and Gesture Recognition, FG 2019. Piscataway : Institute of Electrical and Electronics Engineers, 2019.
@inproceedings{b3ae87bb85274761af45724a07ae7b2f,
title = "Micro-expression detection in long videos using optical flow and recurrent neural networks",
abstract = "Facial micro-expressions are subtle and involuntary expressions that can reveal concealed emotions. Microexpressions are an invaluable source of information in application domains such as lie detection, mental health, sentiment analysis and more. One of the biggest challenges in this field of research is the small amount of available spontaneous micro-expression data. However, spontaneous data collection is burdened by time-consuming and expensive annotation. Hence, methods are needed which can reduce the amount of data that annotators have to review. This paper presents a novel micro-expression spotting method using a recurrent neural network (RNN) on optical flow features. We extract Histogram of Oriented Optical Flow (HOOF) features to encode the temporal changes in selected face regions. Finally, the RNN spots short intervals which are likely to contain occurrences of relevant facial micro-movements. The proposed method is evaluated on the SAMM database. Any chance of subject bias is eliminated by training the RNN using Leave-One-Subject-Out cross-validation. Comparing the spotted intervals with the labeled data shows that the method produced 1569 false positives while obtaining a recall of 0.4654. The initial results show that the proposed method would reduce the video length by a factor of 3.5, while still retaining almost half of the relevant micro-movements. Lastly, as the model gets more data, it becomes better at detecting intervals, which the proposed method suitable for supporting the annotation process.",
author = "Michiel Verburg and Vlado Menkovski",
year = "2019",
month = "5",
day = "1",
doi = "10.1109/FG.2019.8756588",
language = "English",
booktitle = "14th IEEE International Conference on Automatic Face and Gesture Recognition, FG 2019",
publisher = "Institute of Electrical and Electronics Engineers",
address = "United States",

}

Verburg, M & Menkovski, V 2019, Micro-expression detection in long videos using optical flow and recurrent neural networks. in 14th IEEE International Conference on Automatic Face and Gesture Recognition, FG 2019., 8756588, Institute of Electrical and Electronics Engineers, Piscataway, 14th IEEE International Conference on Automatic Face and Gesture Recognition, FG 2019, Lille, France, 14/05/19. https://doi.org/10.1109/FG.2019.8756588

Micro-expression detection in long videos using optical flow and recurrent neural networks. / Verburg, Michiel; Menkovski, Vlado.

14th IEEE International Conference on Automatic Face and Gesture Recognition, FG 2019. Piscataway : Institute of Electrical and Electronics Engineers, 2019. 8756588.

Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

TY - GEN

T1 - Micro-expression detection in long videos using optical flow and recurrent neural networks

AU - Verburg, Michiel

AU - Menkovski, Vlado

PY - 2019/5/1

Y1 - 2019/5/1

N2 - Facial micro-expressions are subtle and involuntary expressions that can reveal concealed emotions. Microexpressions are an invaluable source of information in application domains such as lie detection, mental health, sentiment analysis and more. One of the biggest challenges in this field of research is the small amount of available spontaneous micro-expression data. However, spontaneous data collection is burdened by time-consuming and expensive annotation. Hence, methods are needed which can reduce the amount of data that annotators have to review. This paper presents a novel micro-expression spotting method using a recurrent neural network (RNN) on optical flow features. We extract Histogram of Oriented Optical Flow (HOOF) features to encode the temporal changes in selected face regions. Finally, the RNN spots short intervals which are likely to contain occurrences of relevant facial micro-movements. The proposed method is evaluated on the SAMM database. Any chance of subject bias is eliminated by training the RNN using Leave-One-Subject-Out cross-validation. Comparing the spotted intervals with the labeled data shows that the method produced 1569 false positives while obtaining a recall of 0.4654. The initial results show that the proposed method would reduce the video length by a factor of 3.5, while still retaining almost half of the relevant micro-movements. Lastly, as the model gets more data, it becomes better at detecting intervals, which the proposed method suitable for supporting the annotation process.

AB - Facial micro-expressions are subtle and involuntary expressions that can reveal concealed emotions. Microexpressions are an invaluable source of information in application domains such as lie detection, mental health, sentiment analysis and more. One of the biggest challenges in this field of research is the small amount of available spontaneous micro-expression data. However, spontaneous data collection is burdened by time-consuming and expensive annotation. Hence, methods are needed which can reduce the amount of data that annotators have to review. This paper presents a novel micro-expression spotting method using a recurrent neural network (RNN) on optical flow features. We extract Histogram of Oriented Optical Flow (HOOF) features to encode the temporal changes in selected face regions. Finally, the RNN spots short intervals which are likely to contain occurrences of relevant facial micro-movements. The proposed method is evaluated on the SAMM database. Any chance of subject bias is eliminated by training the RNN using Leave-One-Subject-Out cross-validation. Comparing the spotted intervals with the labeled data shows that the method produced 1569 false positives while obtaining a recall of 0.4654. The initial results show that the proposed method would reduce the video length by a factor of 3.5, while still retaining almost half of the relevant micro-movements. Lastly, as the model gets more data, it becomes better at detecting intervals, which the proposed method suitable for supporting the annotation process.

UR - http://www.scopus.com/inward/record.url?scp=85070447759&partnerID=8YFLogxK

U2 - 10.1109/FG.2019.8756588

DO - 10.1109/FG.2019.8756588

M3 - Conference contribution

AN - SCOPUS:85070447759

BT - 14th IEEE International Conference on Automatic Face and Gesture Recognition, FG 2019

PB - Institute of Electrical and Electronics Engineers

CY - Piscataway

ER -

Verburg M, Menkovski V. Micro-expression detection in long videos using optical flow and recurrent neural networks. In 14th IEEE International Conference on Automatic Face and Gesture Recognition, FG 2019. Piscataway: Institute of Electrical and Electronics Engineers. 2019. 8756588 https://doi.org/10.1109/FG.2019.8756588