The “Black Box” of User Engagement

Measuring user engagement is central to the success of all digital technologies, from social media to search engines, online learning platforms to health and well-being applications. Much of this measurement has focused on analytic metrics, such as dwell time, page visits, and mouse clicks, with the idea that more user activity equates to greater user engagement. However, research has shown that similar behaviours – and physiological reactions – are evoked in users for both positive and negative experiences, making it challenging to disambiguate engagement from frustration. Self-report measures, sometimes criticized for being subjective and lacking in rigour, are largely misunderstood in terms of their ability to open the black box of user experience. The User Engagement Scale, published in 2010 and revised in 2018, has been widely used by industry and academic research teams internationally to evaluate engagement in a variety of settings and with a range of technologies. Questionnaires, when evaluated for reliability and validity, have the potential to help researchers and developers make sense of analytic and neurophysiological measures. Drawing upon over 10 years of studies in the domain of information search, this talk will discuss how multi-method approaches can bring a holistic perspective to the measurement of user engagement.