Single-Item Reliability for Intensive Longitudinal Data: A Three-Level Linear Mixed-Effect Autoregressive Modeling Approach

With the advancement of technology, intensive longitudinal data (ILD) are increasingly collected to study dynamic, moment-to-moment psychological and physiological processes. Despite their widespread use, the issues of measurement error and reliability are often inadequately addressed, particularly in the presence of serial correlation and complex multilevel structures. This article introduces a three-level linear mixed-effects model that explicitly accounts for nested structures (moments within days within persons) and incorporates two distinct autoregressive processes for moment- and day-level serial correlations, partitioning measurement error from within-person variability. Building on this model, we delineate two approaches to defining reliability: a variance partitioning approach and a correlation approach. This framework yields a multifaceted set of reliability coefficients, enabling researchers to distinguish between-person from within-person reliability, and to assess measurement agreement across different time scales (e.g., moment-to-moment and day-to-day). We further discuss the relationship between the proposed reliability measures and those derived from simpler models. We apply our model to study the reliability of a single negative affect item in an ILD study in which students were measured multiple times a day for several days before and after receiving their exam results.