Micro-expression detection in long videos using optical flow and recurrent neural networks

Michiel Verburg, Vlado Menkovski

Onderzoeksoutput: Hoofdstuk in Boek/Rapport/CongresprocedureConferentiebijdrageAcademicpeer review

1 Citaat (Scopus)
1 Downloads (Pure)

Uittreksel

Facial micro-expressions are subtle and involuntary expressions that can reveal concealed emotions. Microexpressions are an invaluable source of information in application domains such as lie detection, mental health, sentiment analysis and more. One of the biggest challenges in this field of research is the small amount of available spontaneous micro-expression data. However, spontaneous data collection is burdened by time-consuming and expensive annotation. Hence, methods are needed which can reduce the amount of data that annotators have to review. This paper presents a novel micro-expression spotting method using a recurrent neural network (RNN) on optical flow features. We extract Histogram of Oriented Optical Flow (HOOF) features to encode the temporal changes in selected face regions. Finally, the RNN spots short intervals which are likely to contain occurrences of relevant facial micro-movements. The proposed method is evaluated on the SAMM database. Any chance of subject bias is eliminated by training the RNN using Leave-One-Subject-Out cross-validation. Comparing the spotted intervals with the labeled data shows that the method produced 1569 false positives while obtaining a recall of 0.4654. The initial results show that the proposed method would reduce the video length by a factor of 3.5, while still retaining almost half of the relevant micro-movements. Lastly, as the model gets more data, it becomes better at detecting intervals, which the proposed method suitable for supporting the annotation process.

Originele taal-2Engels
Titel14th IEEE International Conference on Automatic Face and Gesture Recognition, FG 2019
Plaats van productiePiscataway
UitgeverijInstitute of Electrical and Electronics Engineers
Aantal pagina's6
ISBN van elektronische versie978-1-7281-0089-0
DOI's
StatusGepubliceerd - 1 mei 2019
Evenement14th IEEE International Conference on Automatic Face and Gesture Recognition, FG 2019 - Lille, Frankrijk
Duur: 14 mei 201918 mei 2019

Congres

Congres14th IEEE International Conference on Automatic Face and Gesture Recognition, FG 2019
LandFrankrijk
StadLille
Periode14/05/1918/05/19

Vingerafdruk

Optical flows
Recurrent neural networks
Health

Citeer dit

Verburg, M., & Menkovski, V. (2019). Micro-expression detection in long videos using optical flow and recurrent neural networks. In 14th IEEE International Conference on Automatic Face and Gesture Recognition, FG 2019 [8756588] Piscataway: Institute of Electrical and Electronics Engineers. https://doi.org/10.1109/FG.2019.8756588
Verburg, Michiel ; Menkovski, Vlado. / Micro-expression detection in long videos using optical flow and recurrent neural networks. 14th IEEE International Conference on Automatic Face and Gesture Recognition, FG 2019. Piscataway : Institute of Electrical and Electronics Engineers, 2019.
@inproceedings{b3ae87bb85274761af45724a07ae7b2f,
title = "Micro-expression detection in long videos using optical flow and recurrent neural networks",
abstract = "Facial micro-expressions are subtle and involuntary expressions that can reveal concealed emotions. Microexpressions are an invaluable source of information in application domains such as lie detection, mental health, sentiment analysis and more. One of the biggest challenges in this field of research is the small amount of available spontaneous micro-expression data. However, spontaneous data collection is burdened by time-consuming and expensive annotation. Hence, methods are needed which can reduce the amount of data that annotators have to review. This paper presents a novel micro-expression spotting method using a recurrent neural network (RNN) on optical flow features. We extract Histogram of Oriented Optical Flow (HOOF) features to encode the temporal changes in selected face regions. Finally, the RNN spots short intervals which are likely to contain occurrences of relevant facial micro-movements. The proposed method is evaluated on the SAMM database. Any chance of subject bias is eliminated by training the RNN using Leave-One-Subject-Out cross-validation. Comparing the spotted intervals with the labeled data shows that the method produced 1569 false positives while obtaining a recall of 0.4654. The initial results show that the proposed method would reduce the video length by a factor of 3.5, while still retaining almost half of the relevant micro-movements. Lastly, as the model gets more data, it becomes better at detecting intervals, which the proposed method suitable for supporting the annotation process.",
author = "Michiel Verburg and Vlado Menkovski",
year = "2019",
month = "5",
day = "1",
doi = "10.1109/FG.2019.8756588",
language = "English",
booktitle = "14th IEEE International Conference on Automatic Face and Gesture Recognition, FG 2019",
publisher = "Institute of Electrical and Electronics Engineers",
address = "United States",

}

Verburg, M & Menkovski, V 2019, Micro-expression detection in long videos using optical flow and recurrent neural networks. in 14th IEEE International Conference on Automatic Face and Gesture Recognition, FG 2019., 8756588, Institute of Electrical and Electronics Engineers, Piscataway, Lille, Frankrijk, 14/05/19. https://doi.org/10.1109/FG.2019.8756588

Micro-expression detection in long videos using optical flow and recurrent neural networks. / Verburg, Michiel; Menkovski, Vlado.

14th IEEE International Conference on Automatic Face and Gesture Recognition, FG 2019. Piscataway : Institute of Electrical and Electronics Engineers, 2019. 8756588.

Onderzoeksoutput: Hoofdstuk in Boek/Rapport/CongresprocedureConferentiebijdrageAcademicpeer review

TY - GEN

T1 - Micro-expression detection in long videos using optical flow and recurrent neural networks

AU - Verburg, Michiel

AU - Menkovski, Vlado

PY - 2019/5/1

Y1 - 2019/5/1

N2 - Facial micro-expressions are subtle and involuntary expressions that can reveal concealed emotions. Microexpressions are an invaluable source of information in application domains such as lie detection, mental health, sentiment analysis and more. One of the biggest challenges in this field of research is the small amount of available spontaneous micro-expression data. However, spontaneous data collection is burdened by time-consuming and expensive annotation. Hence, methods are needed which can reduce the amount of data that annotators have to review. This paper presents a novel micro-expression spotting method using a recurrent neural network (RNN) on optical flow features. We extract Histogram of Oriented Optical Flow (HOOF) features to encode the temporal changes in selected face regions. Finally, the RNN spots short intervals which are likely to contain occurrences of relevant facial micro-movements. The proposed method is evaluated on the SAMM database. Any chance of subject bias is eliminated by training the RNN using Leave-One-Subject-Out cross-validation. Comparing the spotted intervals with the labeled data shows that the method produced 1569 false positives while obtaining a recall of 0.4654. The initial results show that the proposed method would reduce the video length by a factor of 3.5, while still retaining almost half of the relevant micro-movements. Lastly, as the model gets more data, it becomes better at detecting intervals, which the proposed method suitable for supporting the annotation process.

AB - Facial micro-expressions are subtle and involuntary expressions that can reveal concealed emotions. Microexpressions are an invaluable source of information in application domains such as lie detection, mental health, sentiment analysis and more. One of the biggest challenges in this field of research is the small amount of available spontaneous micro-expression data. However, spontaneous data collection is burdened by time-consuming and expensive annotation. Hence, methods are needed which can reduce the amount of data that annotators have to review. This paper presents a novel micro-expression spotting method using a recurrent neural network (RNN) on optical flow features. We extract Histogram of Oriented Optical Flow (HOOF) features to encode the temporal changes in selected face regions. Finally, the RNN spots short intervals which are likely to contain occurrences of relevant facial micro-movements. The proposed method is evaluated on the SAMM database. Any chance of subject bias is eliminated by training the RNN using Leave-One-Subject-Out cross-validation. Comparing the spotted intervals with the labeled data shows that the method produced 1569 false positives while obtaining a recall of 0.4654. The initial results show that the proposed method would reduce the video length by a factor of 3.5, while still retaining almost half of the relevant micro-movements. Lastly, as the model gets more data, it becomes better at detecting intervals, which the proposed method suitable for supporting the annotation process.

UR - http://www.scopus.com/inward/record.url?scp=85070447759&partnerID=8YFLogxK

U2 - 10.1109/FG.2019.8756588

DO - 10.1109/FG.2019.8756588

M3 - Conference contribution

AN - SCOPUS:85070447759

BT - 14th IEEE International Conference on Automatic Face and Gesture Recognition, FG 2019

PB - Institute of Electrical and Electronics Engineers

CY - Piscataway

ER -

Verburg M, Menkovski V. Micro-expression detection in long videos using optical flow and recurrent neural networks. In 14th IEEE International Conference on Automatic Face and Gesture Recognition, FG 2019. Piscataway: Institute of Electrical and Electronics Engineers. 2019. 8756588 https://doi.org/10.1109/FG.2019.8756588