Context-Enhanced Human-Robot Interaction: Exploring the Role of System Interactivity and Multimodal Stimuli on the Engagement of People with Dementia

Yuan Feng (Corresponding author), G. Perugia, Suihuai Yu, Emilia I. Barakova, Jun Hu, G.W.M. (Matthias) Rauterberg

Research output: Contribution to journalArticleAcademicpeer-review

12 Citations (Scopus)
267 Downloads (Pure)


Engaging people with dementia (PWD) in meaningful activities is the key to promote their quality of life. Design towards a higher level of user engagement has been extensively studied within the human-computer interaction community, however, few extend to PWD. It is generally considered that increased richness of experiences can lead to enhanced engagement. Therefore, this paper explores the effects of rich interaction in terms of the role of system interactivity and multimodal stimuli by engaging participants in context-enhanced human-robot interaction activities. The interaction with a social robot was considered context-enhanced due to the additional responsive sensory feedback from an augmented reality display. A field study was conducted in a Dutch nursing home with 16 residents. The study followed a two by two mixed factorial design with one within-subject variable - multimodal stimuli - and one between-subject variable - system interactivity. A mixed method of video coding analysis and observational rating scales was adopted to assess user engagement comprehensively. Results disclose that when additional auditory modality was included besides the visual-tactile stimuli, participants had significantly higher scores on attitude, more positive behavioral engagement during activity, and a higher percentage of communications displayed. The multimodal stimuli also promoted social interaction between participants and the facilitator. The findings provide sufficient evidence regarding the significant role of multimodal stimuli in promoting PWD’s engagement, which could be potentially used as a motivation strategy in future research to improve emotional aspects of activity-related engagement and social interaction with the human partner.

Original languageEnglish
Pages (from-to)807-826
Number of pages20
JournalInternational Journal of Social Robotics
Issue number3
Early online date9 Oct 2021
Publication statusPublished - Apr 2022

Bibliographical note

Funding Information:
The authors would like to acknowledge all participated residents, caregivers and Trudy van de schans-Robijns from Vitalis for their involvement and contributions. We also thank Famke Boschman for her help in the user study and data analysis. This study is partly sponsored by China Scholarship Council.


  • People with dementia
  • Interactivity
  • Multimodal stimuli
  • Engagement
  • Social robots
  • Context-enhanced human-robot interaction


Dive into the research topics of 'Context-Enhanced Human-Robot Interaction: Exploring the Role of System Interactivity and Multimodal Stimuli on the Engagement of People with Dementia'. Together they form a unique fingerprint.

Cite this