Comparative Analysis of AI, Human, and Self-Service-Technology Systems in Digital Service Experiences: An Explainable Artificial Intelligence Approach

Evaluating the user experience of complex, large-scale socio-technical systems is a significant data-science challenge, as traditional survey methods often fail to capture authentic user perceptions. This study proposes and validates a novel computational framework that combines Structural Topic Modeling (STM) and SHapley Additive exPlanations (SHAP) to mine and interpret unstructured user-generated data at scale.We apply this framework to comparatively analyze consumer satisfaction with three dominant service modalities: AI-powered agents, human service providers, and standardized self-service technology (SST). Analyzing a large-scale dataset of 260,577 user reviews, our approach first uses STM to discover 100 distinct dimensions of the user experience, which are then mapped to an integrated Technology-System-User (TSU) theoretical model. Second, an XGBoost model predicts satisfaction, and SHAP analysis is used to identify the precise drivers of satisfaction for each modality. The findings reveal three divergent paths to a positive user experience: human-led services rely on interpersonal connection, AI services on a strong price-value proposition and a non-judgmental safe space, and standardized systems on gamified engagement. This study contributes a robust, interpretable, and scalable methodological framework for analyzing user feedback, offering actionable, data-driven insights for the design and optimization of complex, human-in-the-loop, and AI-powered digital systems.