End-to-End Multi-Modal Behavioral Context Recognition in a Real-Life Setting

Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

1 Citation (Scopus)
40 Downloads (Pure)


Smart devices of everyday use (such as smartphones and wearables) are increasingly integrated with sensors that provide immense amounts of information about a person's daily life. The automatic and unobtrusive sensing of human behavioral context can help develop solutions for assisted living, fitness tracking, sleep monitoring, and several other fields. Towards addressing this issue, we raise the question: can a machine learn to recognize a diverse set of contexts and activities in a real-life through jointly learning from raw multi-modal signals (e.g., accelerometer, gyroscope and audio)? In this paper, we propose a multi-stream network comprising of temporal convolution and fully-connected layers to address the problem of multi-label behavioral context recognition. A four-stream network architecture handles learning from each modality with a contextualization module which incorporates extracted representations to infer a user's context. Our empirical evaluation suggests that a deep convolutional network trained end-to-end achieves comparable performance to manual feature engineering with minimal effort. Furthermore, the presented architecture can be extended to include similar sensors for performance improvements and handles missing modalities through multi-task learning on a highly imbalanced and sparsely labeled dataset.

Original languageEnglish
Title of host publicationFUSION 2019 - 22nd International Conference on Information Fusion
Place of PublicationPiscataway
PublisherInstitute of Electrical and Electronics Engineers
Number of pages9
ISBN (Electronic)9780996452786
Publication statusPublished - Jul 2019
Event22nd International Conference on Information Fusion, FUSION 2019 - Ottawa, Canada
Duration: 2 Jul 20195 Jul 2019


Conference22nd International Conference on Information Fusion, FUSION 2019


  • activity detection
  • context recognition
  • deep learning
  • multi-label
  • multi-modal data
  • multi-task
  • temporal convolution network


Dive into the research topics of 'End-to-End Multi-Modal Behavioral Context Recognition in a Real-Life Setting'. Together they form a unique fingerprint.

Cite this