Meet your users in situ data collection from within apps in large-scale deployments

Increasingly, ‘app-store' releases of software are used as a vehicle for large-scale user trials ‘in the wild'. Several opportunities and methodological challenges arise from having little or no access to users, other than through the application itself. So far, researchers have needed to hardcode survey items into the software application studied, which is laborious and error prone. This paper discusses how these problems are addressed using TEMPEST, a platform for longitudinal in situ data collection. The authors illustrate the use of TEMPEST to study the deployment and real-world use of a tablet application called idAnimate; this application has been designed to support the creation of simple animations as design representations during the creative design process. The authors discuss how the tool has supported the gathering of data in over 4000 installations, both from a development and research perspective, and relate their experiences to current research perspectives on large-scale app trials.