Impact of Imaging Acquisition and Protocol Variability on Artificial Intelligence Model Performance: A Secondary Analysis of the ASFNR Artificial Intelligence Competition

BACKGROUND AND PURPOSE: Artificial intelligence (AI) models have shown promise in neuroradiology, yet their real-world generalizability remains uncertain, partly due to variability in imaging acquisition and protocols. We aimed to evaluate the impact of data source, scanner manufacturer, scan mode, slice thickness, and the AI models, developed by participating teams, on AI performance in this secondary analysis of the 2019 American Society of Functional Neuroradiology (ASFNR) AI Competition. MATERIALS AND METHODS: We included 1177 anonymized noncontrast head CT scans from 5 institutions. Four teams participated, developing models to detect acute ischemic stroke, intracranial hemorrhage, and mass effect and to assess age-appropriate normality. Generalized estimating equations were used to evaluate the effects of the variables on model performance, and collinearity diagnostics were applied to exclude redundant variables. RESULTS: Due to collinearity with the scanner manufacturer, data source was excluded from the model. Across all tasks, the AI model significantly influenced the performance. The scanner manufacturer was significantly associated with accuracy in detecting intracranial hemorrhage and acute ischemic stroke but not mass effect or age-based normality. Slice thickness was significantly associated with detection of intracranial hemorrhage and mass effect, with thinner slices yielding higher accuracy, but it showed no effect on ischemic stroke or normality assessments. The scan mode did not significantly influence performance for any task. CONCLUSIONS: This secondary analysis demonstrates that imaging acquisition and protocol variability may significantly affect AI model performance. Scanner manufacturer, slice thickness, and the developed AI model were significantly associated with model accuracy, whereas scan mode had no significant impact. Among these, the developed AI model consistently proved the most influential, reflecting the importance of training data, model architecture, and preprocessing methods.