Lab testing beyond usability: challenges and recommendations for assessing user experiences

In the “third wave” of human-computer interaction (HCI), the advent of the conceptual approach of UX broadens and changes the HCI landscape. Methods approved before, mainly within the conceptual approach of usability, are still widely used, and yet their adequacy for UX evaluation remains uncertain in many applications. Laboratory testing is undoubtedly the most prominent example of such a method. Hence, in this study, we investigated how the more comprehensive and emotional scope of UX can be assessed by laboratory testing.

In this paper, we report on a use case study involving 70 participants. They first took part in user/laboratory tests and then were asked to evaluate their experience with the two systems (perceived UX) by filling out an AttrakDiff scale and a UX needs fulfillment questionnaire. We conducted post-test interviews to better understand participants’ experiences. We analyzed how the participants’ perceived UX depends on quantitative (e.g., task completion time, task sequence, level of familiarity with the system) and qualitative aspects (think aloud, debriefing interviews) within the laboratory context.

Results indicate that the laboratory setting has a strong impact on the participants’ perceived UX, and support a discussion of the quality and limitations of laboratory evaluations regarding UX assessment. In this paper, we have identified concrete challenges and have provided solutions and tips useful for both practitioners and researchers who seek to account for the subjective, situated, and temporal nature of the UX in their assessments.