Enumerator Honesty and Survey Data Quality Experimental Evidence from Large Household Surveys in Ethiopia

Abstract High-stakes household and firm surveys are central to empirical research and policy analysis, especially in settings with limited administrative data. However, the quality of survey data depends not only on survey design but also on how surveys are implemented in the field. Monitoring survey implementation and data collection remains costly and imperfect, especially in remote contexts, implying the disproportionate and critical role of enumerators---the frontline agents who implement surveys---in shaping data quality. This paper examines the implications of enumerator honesty, its predictors, and its implications on survey data quality. We measure enumerator honesty using an incentivized experimental reporting task with a substantial financial stake and link these experimental measures of honesty to data collected by the same enumerators across two large-scale household surveys in Ethiopia. We specifically examine: (i) the distribution of honesty across enumerators, (ii) whether experimentally measured honesty predicts enumerator behavior and decisions, and (iii) whether honesty is associated with pre-defined indicators of survey data quality. To understand what predicts enumerator honesty, we aim to use machine learning methods and analyze rich data on enumerators' socioeconomic backgrounds, economic preferences, personality traits, socioemotional, and cognitive ability. By linking experimentally measured honesty to real survey-related outcomes across multiple surveys, the study aims to provide actionable evidence to inform enumerator recruitment in large-scale field studies and, hence, ultimately improve survey data quality. Interventions The study is being conducted in conjunction with ongoing and forthcoming large-scale household surveys in Ethiopia. The surveys cover multiple regions of the country and will be implemented over several weeks by a team of trained enumerators and field supervisors. The scale and structure of the survey, including its spatial and temporal variation and the number of enumerators involved, provide a unique opportunity to measure enumerator honesty and the extent it predicts survey data quality. To measure enumerator honesty, all enumerators and supervisors will participate in an incentivized reporting task drawn from the experimental literature on dishonesty (Fischbacher and F¨ollmi-Heusi, 2013; Barfort et al., 2019). The task consists of 30 independent rounds of a coin-tossing game. The coin tossing exercise uses a coin with two sides: Lion and Weighing scale. The cumulative monetary payments increase with the number of reported Lion. Because the outcomes of the coin tosses are unobserved by the researchers, participants have the opportunity to misreport their success in reporting the outcomes. Several studies have demonstrated that dis(honesty) measured through this process predicts actual unethical behavior and corruption (Fischbacher and F¨ollmi-Heusi, 2013; Barfort et al., 2019; Olsen et al., 2019; Cohn and Mar´echal, 2018). Enumerator honesty will be measured by comparing the distribution of reported correct predictions to the distribution implied by truthful reporting under random chance. Deviations from the expected distribution will be used to construct an individual-level measure of each enumerator’s propensity to misreport. Intervention: Measuring survey enumerators’ honesty The household surveys are being (will be) conducted over several weeks. The honesty experiment will be administered to all enumerators and supervisors at the end of the household surveys. All experimental sessions will be conducted in a standardized manner and closely monitored to ensure adherence to the protocol and consistency across participants. The tossing exercise uses an Ethiopian 1 Birr Coin (Birr is Ethiopian currency and at the time of the survey, 1 USD=155 Birr), which has two sides: Lion and Weighing scale. In each round, participants privately toss a coin and report the realized outcome, with a higher payoff for reporting Lion and a lower payoff for reporting tails. Primary Outcomes 1. Enumerator's real-world preferences and tendencies for unethical behavior 2. Drop-out rates 3. Interview duration 4. Distance covered by enumerators within each village to capture enumerator's effort. 5. Number of affirmative responses in roaster items (e.g., food and non-food items; livestock species) 6. Share of reported missing households or non-consented households. 7. Objectively measured performance reports by auditors and supervisors. Primary Outcomes (Explanation) 1. Unethical preferences and actions: we aim to elicit enumerators’ tendencies to pursue unethical decisions, including operating without a formal license, avoiding tax payments, and reporting moral values associated with paying bribes. For this purpose, we ask enumerators about their formal license to operate as well as their tax payment history. Similarly, we collect their moral values associated with paying bribes and avoiding fare on public transport. 2. Dropout rates - We measure dropout rates using the participation and dropout rates of enumerators. Enumerators can drop out for several reasons, including because of supervisor’s and survey firm’s decision related to their performance. For each enumerator, we observe whether they finish the task they were assigned or whether they have dropped out of the survey because of the survey firm’s decision or their own discretion. 3. Interview duration - survey timestamps recorded at each module - measures the time taken (in minutes) to complete specific survey modules, such as consumption, livestock, and household roster sections. Module-level duration is used to assess enumerator effort, fatigue, and potential data quality concerns, as unusually short or long completion times may indicate skipping, rushing, or respondent disengagement. 4. Distance covered by enumerators within each village - We aim to assemble the distance covered by each enumerator in each village using geo-reference of survey respondents’ (household) residence. This helps us to capture the level of effort exerted by each enumerator. For this purpose, we aim to compute the Euclidean distance between respondents and consider various forms of aggregation of these distances within each village. 5. Number of affirmative responses in roasters. This outcome captures the total number of food and non-food items for which a household reports positive consumption or expenditure during the reference period. In the survey design, enumerators first record a binary yes/no response indicating whether the household consumed or purchased each listed item, and detailed follow-up questions are asked only if “yes” is selected. Unusually low number of reported items may indicate deliberate under-reporting by enumerators to reduce interview length and effort. Each additional item marked as “yes” increases. interview length and cognitive effort for both the enumerator and the respondent. Beyond household consumption expenditure, we also consider livestock spices reported. This outcome measures the total number of distinct livestock types (e.g., cattle, sheep, goats, poultry, etc.) reported as owned by the household. Similar to the expenditure module, the livestock section is programmed using a screening question structure: enumerators first record whether the household owns each type of livestock, and detailed follow-up questions, such as herd size, sales, and losses, are asked only for livestock types reported as owned. 6. Share of missing (non-consented) households: This outcome measures the number of households recorded as unavailable, absent, or not found during fieldwork. Relatively large share of missing households may signal systematic avoidance behavior, including if they are located too far from village centers. 7. Objectively measured performance reports by auditors and supervisors: We administer a short monitoring survey to evaluate the performance of enumerators by those who monitor the data from the backend. For each enumerator, we ask their corresponding supervisors and auditors to score them based on the perceived level of effort and quality of data. They score these enumerators based on random audio audits and callbacks, as well as interactions with the enumerators. Secondary Outcomes 1. Deviation of household economic ranking from rankings based on welfare (consumption) measures. 2. The household ranking versus objective measures of poverty. Experimental Design Measuring enumerator honesty To measure enumerator honesty, we will conduct a lab-in-the-field experiment with enumerators involved in collecting household surveys in Ethiopia. Approximately 200 enumerators, including supervisors, will participate in the experiment. Participants will be informed that the task is designed to understand decision-making among enumerators engaged in data collection activities and to test how well they guess in situations characterized by randomness. They will also be informed that they earn money based on their reported guesses. Participation takes approximately one hour, with potential earnings of up to 1500 Birr. The design follows the approach of Barfort et al. (2019) and Fischbacher and F¨ollmi-Heusi (2013) to measure enumerators’ propensity for dishonesty, adapted to the survey context. For logistical simplicity, we will implement a coin-tossing task instead of a dice-rolling task, as in related studies (e.g., Abeler et al., 2014; Cohn et al., 2014; Janezic and Gallego, 2020). Participants will be instructed to use a coin of the standard local currency in Ethiopia. Before the main task, participants will complete six practice rounds to familiarize themselves with the procedure. In the main task, participants will complete 30 independent rounds. In each round, participants privately toss the coin and subsequently report the realized outcome. Reporting heads yields a payment of 50 Birr, while reporti