Balancing Cost and Response in Longitudinal Surveys: Evidence from a Choice+ Mode Experiment

In longitudinal studies, retaining participants over time is necessary to ensure sufficient sample sizes for analyses and reduce the potential for bias; however, efforts to improve response rates can increase the cost of data collection substantially. To explore methods that would best balance data quality and cost for a longitudinal study of veterans deployed during the Gulf War era, we conducted an experiment comparing two multi-mode designs which offered a higher incentive for completing the survey by web. Veterans were randomly assigned to receive one of two protocols: web offered initially with paper and the bonus web incentive introduced later (sequential Choice+), or both web and paper offered at the same time, with the bonus web incentive offered at the beginning (concurrent Choice+). Both protocols offered Computer Assisted Telephone Interviews to non-respondents. We examined the impact of each approach on response rates, sample representativeness, and cost. The response rate for the concurrent Choice+ group was 2.8 percentage points above that of the sequential Choice+ group (47.5 versus 44.7 percent); however, there were no differences in the characteristics of respondents in each group compared with the eligible sample or between respondents in each group on key survey items, an indicator that lower response did not result in biased estimates. The cost to implement the concurrent Choice+ design was substantially higher. These findings suggest that the less costly sequential Choice+ approach may provide a better tradeoff between quality and cost, particularly for longitudinal studies seeking cost efficiencies. Future research may benefit from exploring varying amounts for the web bonus and how a sequential Choice+ protocol could be used to increase response rates for multimode surveys that include web and CATI but not paper.