The Distracted Participant? Experience Sampling Response Behavior and Participant Disturbance in Social Situations

Research using self-reports hinges on the assumption that participants pay sufficient attention to questionnaires to provide valid data. This assumption is particularly tenuous in experience sampling method (ESM) studies, where participants complete questionnaires in daily life across a range of potentially distracting situations. Previous research suggests that participants may be particularly distracted when responding to ESM questionnaires in social situations, especially when engaging in social interactions. Yet, the effects of these environmental distractions on response behavior and, consequently, data quality remain poorly understood. We investigated the effects of distracting environments on disturbance and response behavior across various social and non-social situations. ESM data from three young adult samples (combined N = 293) and a general population youth sample (N = 1903) was analyzed with multilevel (logistic) regressions. In line with previous research, adults were significantly more disturbed by assessments when in company compared to when alone, especially when also interacting with their company. In addition, we found small but significant differences in response behavior between social settings in adults, with changes pointing towards lower data quality when in company. Interestingly, patterns were different–in some cases even reversed–in school-going adolescents. While our findings suggest that the distraction of social settings affects participant burden and response behavior, the influence on data quality was minor. Differences across samples suggest that the setting of the social experience (in vs outside school) needs to be considered. Preparing participants for sampling in distracting (social) environments may help safeguard data quality and reduce participant burden.