Trust as Multiplicative Self-Calibration: A Pre-Registered Secondary Analysis of Dunning-Kruger Effects on AI Reliance (He et al., CHI 2023)

Read full text Summarise with VIRAT
Authors: Christopher Byrd
Cited by: 0

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

# OSF Pre-Registration: Ready-to-Paste Document **Study Title:** Trust as Multiplicative Self-Calibration: A Pre-Registered Secondary Analysis of Dunning-Kruger Effects on AI Reliance (He et al., CHI 2023) **OSF Template:** Standard Pre-Registration (AsPredicted or OSF Prereg) **Status:** READY TO PASTE. Do not run any analyses before locking this registration. --- --- ## 1. STUDY INFORMATION ### Title Trust as Multiplicative Self-Calibration: A Pre-Registered Secondary Analysis of Dunning-Kruger Effects on AI Reliance ### Authors Christopher Byrd, Lyrastone ### Description This is a pre-registered secondary analysis of publicly available data from He, Kuiper, and Gadiraju (2023), "Who Should I Trust: AI or Myself? Leveraging Human and AI Correctness Likelihood for Automation Bias Mitigation," presented at CHI 2023. The original dataset is publicly available at https://github.com/RichardHGL/CHI2023_DKE under a research-use license. We apply the Trust Flow Delegation (TFD) framework to reinterpret the original study's Dunning-Kruger effect findings. TFD proposes that AI reliance is a multiplicative function of self-trust calibration and AI accuracy, rather than an additive combination. The original study found that participants who overestimate their task performance (overestimators) exhibit lower appropriate reliance on an AI advisor. TFD reinterprets this as a self-trust inflation mechanism: when self-assessed accuracy is inflated relative to actual accuracy, the trust function's output is suppressed regardless of the AI's objective accuracy on any given task. The design contribution framing for this work is Positive Double-Loop Learning (PDLL), a design pattern for personal AI governance in which behavioral rules emerge from observed reliance failures and are recursively updated to repair trust calibration. The He et al. dataset operationalizes one PDLL cycle: the tutorial intervention is a designed recalibration mechanism intended to repair overestimators' inflated self-trust. This pre-registration specifies two testable predictions (Prediction 1 and Prediction 3). A third prediction (engagement moderation of the tutorial effect) was considered and is formally withdrawn here because the required variable (per-participant completion time from `demographic.csv`) is not included in the public dataset. See Section 5 (Analysis Plan) for full details. We declare that, at the time of this pre-registration, we have inspected the repository structure and variable names in `all_valid_data.csv` but have NOT computed any reliance measures, cross-tabulated any variables, run any correlations, or fit any models. All reliance-level analysis is deferred until after this registration is locked. ### Hypotheses **Hypothesis 1 (Self-Trust Miscalibration Suppresses AI Reliance):** Participants who overestimate their task performance (self-assessed correct responses exceed actual correct responses, i.e., miscalibration > 0) will show significantly lower appropriate AI reliance compared to accurate assessors (miscalibration = 0) and underestimators (miscalibration < 0). Furthermore, the effect will be graded: the degree of overestimation will be a significant negative predictor of appropriate reliance in continuous regression models, consistent with TFD's multiplicative suppression mechanism (rather than a simple categorical offset, which would be consistent with an additive model). **Hypothesis 2 (Engagement Moderation): FORMALLY WITHDRAWN** This hypothesis required per-participant task completion times as an engagement proxy. The variable is not available in the public dataset. This hypothesis is not tested and no result is reported for it. **Hypothesis 3 (Multiplicative Model Outperforms Additive Model):** A logistic regression model in which self-trust miscalibration and AI accuracy enter as a multiplicative interaction term (Model A) will fit observed trial-level AI-following behavior significantly better than an equivalent model in which miscalibration and AI accuracy enter as independent additive terms (Model B), as assessed by AIC, BIC, and a likelihood ratio test. The critical test is whether the miscalibration x AI-accuracy interaction term provides significant incremental fit. --- ## 2. DESIGN PLAN ### Study Type Secondary data analysis. This study involves no new data collection. We analyze an existing, publicly available dataset. ### Study Design He et al. (CHI 2023) conducted a 2 (tutorial: present vs. absent) x 2 (explainable AI: XAI present vs. absent) between-subjects experiment on Prolific. Participants completed 19 logical reasoning tasks in a randomized order. For each task, they first submitted their own answer, then viewed the AI advisor's recommendation, and then submitted a final answer. Three of the 19 tasks were attention checks; 16 tasks formed the behavioral data of interest. The study was divided into two phases of 6 tasks each, separated by buffer tasks. After the first phase, participants self-assessed their performance (how many of the first 6 tasks they answered correctly) and reported their trust in the AI. The tutorial or XAI intervention was delivered between phases. After the second phase, participants again self-assessed and reported trust. The primary independent manipulation relevant to our analysis is the tutorial condition, which provided participants with targeted feedback designed to recalibrate their self-assessments. The XAI condition is included as a covariate and in exploratory analyses. **Important coding note (pre-registered):** The analysis code in `util.py` from the original repository uses an inverted coding scheme for the tutorial and XAI variables: `tutorial = 1` in the raw CSV maps to the "no tutorial" condition, and `tutorial = 0` maps to the "with tutorial" condition. Before any analysis, we will verify this coding by cross-referencing raw cell sizes against the condition sizes reported in the He et al. paper. All analyses will use human-readable condition labels (with/without tutorial), not the raw binary codes. --- ## 3. SAMPLING PLAN ### Existing Data We confirm the following: we have reviewed the repository structure and the column names and value ranges of `all_valid_data.csv`. We have NOT analyzed the data. We have NOT computed reliance measures, cross-tabulations, correlations, or model fits. The data exists in the public repository and was available to us before this pre-registration was written. By registering before analysis, we are adopting the "data exists but has not been analyzed" level of OSF pre-registration. ### Data Collection Procedures Data were collected by He, Kuiper, and Gadiraju (2023) via the Prolific crowdsourcing platform. Participants were recruited in multiple batches, which produced unequal cell sizes across the four experimental conditions. All participants received monetary compensation per Prolific norms. The study was conducted in English. Participants who failed one or more of four embedded attention checks were excluded, yielding a final sample of N = 249. The data are deposited publicly at https://github.com/RichardHGL/CHI2023_DKE under a research-use-only license. ### Sample Size N = 249 participants, each providing 16 task-level observations, yielding approximately 3,984 task-level data points for trial-level analyses (Hypothesis 3). For between-subjects analyses (Hypothesis 1), power depends on the distribution of participants across miscalibration categories, which we have not examined. Based on the original paper's reported findings, all three calibration groups (overestimators, accurate assessors, underestimators) have adequate N for non-parametric group comparison tests. No additional participants will be recruited. The sample size is fixed. --- ## 4. VARIABLES ### Manipulated Variables (from original He et al. experiment) 1. **Tutorial condition** (between-subjects, binary): Presence vs. absence of a tutorial that provided targeted feedback on the AI's reasoning process, designed to recalibrate participant self-assessments. Note: coded inversely in raw data (`tutorial = 1` = no tutorial; see Section 2 for verification procedure). 2. **XAI condition** (between-subjects, binary): Presence vs. absence of explainable AI recommendations that included contrastive explanations for the AI's chosen answer. Note: similarly inverted in raw data. ### Measured Variables 3. **Self-assessed performance** (`surveySelf1`, `surveySelf2`): Participant's estimate of how many of the first (then second) group of 6 tasks they answered correctly. Scale: 0-6 integer. 4. **Self-assessed percentile** (`surveyPercentage1`, `surveyPercentage2`): Participant's estimate of what percentage of other participants they outperformed. Scale: 0-100 continuous. 5. **Trust in automation, first measurement** (`tia1_1`, `tia1_2`): Two-item measure of trust in the AI advisor, collected after the first task group. Items coded 0-4 in raw data; composite shifted to 1-5 scale per original analysis code. 6. **Trust in automation, second measurement** (`tia2_1`, `tia2_2`): Two-item trust measure collected after the second task group. Note: tia2 items range 0-6 in the raw data (7-point scale), which differs from tia1's 5-point scale. We will examine this discrepancy and, if confirmed, rescale tia2 to match tia1 before computing change scores. We will report this issue transparently regardless of what we find. 7. **Propensity to trust** (`pt1`, `pt2`, `pt3`): Three-item dispositional trust scale. Coded 0-4 in raw data; composite shifted to 1-5 scale. Used as covariate. 8. **Affinity for Technology Interaction** (`ati1`-`ati9`): Nine-item ATI scale. Coded 0-5 in raw data; composite shifted to 1-6 scale. Reverse-coded items: 3, 6, 8. Used as covariate. 9. **Initial task decisions** (`question0`-`question17`, less attention check items): Participant's first answer to each task before seeing AI advice. Used to compute init

Fields of Study & Subjects

Human-Automation Interaction and Safety Ethics and Social Impacts of AI Social Robot Interaction and HRI

Publication details

Publication date
1 January 2026
Language
EN
Document type
Preprint