PRESS/HOLD/RELEASE ultrasonic gestures and low complexity recognition based on TCN

Emad Ibrahim, Min Li, Jose Pineda de Gyvez

Onderzoeksoutput: Hoofdstuk in Boek/Rapport/CongresprocedureConferentiebijdrageAcademicpeer review

4 Citaten (Scopus)


Targeting ultrasound-based gesture recognition, this paper proposes a new universal PRESS/HOLD/RELEASE approach that leverages the diversity of gestures performed on smart devices such as mobile phones and IoT nodes. The new set of gestures are generated by interleaving PRESS/HOLD/RELEASE patterns; abbreviated as P/H/R, with gestures like sweeps between a number of microphones. P/H/R patterns are constructed by a hand as it approaches a top of a microphone to generate a virtual Press. After that, the hand settles for an undefined period of time to generate a virtual Hold and finally departs to generate a virtual Release. The same hand can sweep to a 2 nd microphone and perform another P/H/R. Interleaving the P/H/R patterns expands the number of performed gestures. Assuming an on-board speaker transmitting ultrasonic signals, the detection is performed on Doppler shift readings generated by a hand as it approaches and departs a top of a microphone. The Doppler shift readings are presented in a sequence of down-mixed ultrasonic spectrogram frames. We train a Temporal Convolutional Network (TCN) to classify the P/H/R patterns under different environmental noises. Our experimental results show that such P/H/R patterns at a top of a microphone can be achieved with 96.6% accuracy under different noise conditions. A group of P/H/R based gestures has been tested on commercially off-The-shelf (COTS) Samsung Galaxy S7 Edge. Different P/H/R interleaved gestures (such as sweeps, long taps, etc.) are designed using two microphones and a single speaker while using as low as \sim 5\mathrm{K} parameters and as low as \sim 0.15 Million operations (MOPs) in compute power per inference. The P/H/R interleaved set of gestures are intuitive and hence are easy to learn by end users. This paves its way to be deployed by smartphones and smart speakers for mass production.

Originele taal-2Engels
TitelProceedings of SiPS 2019: the IEEE International Workshop on Signal Processing Systems
Plaats van productiePiscataway
UitgeverijInstitute of Electrical and Electronics Engineers
Aantal pagina's6
ISBN van elektronische versie9781728119274
StatusGepubliceerd - okt. 2019
Evenement33rd IEEE Workshop on Signal Processing Systems, SiPS 2019 - Nanjing, China
Duur: 20 okt. 201923 okt. 2019
Congresnummer: 33


Congres33rd IEEE Workshop on Signal Processing Systems, SiPS 2019
Verkorte titelSiPS 2019


Duik in de onderzoeksthema's van 'PRESS/HOLD/RELEASE ultrasonic gestures and low complexity recognition based on TCN'. Samen vormen ze een unieke vingerafdruk.

Citeer dit