Performance and Interpretability Trade-Offs in Reinforcement Learning for Sepsis Treatment: Comparing Offline and Online Approaches

Background/Objectives: Sepsis is a leading cause of mortality in critical care and is associated with substantial long-term neurological morbidity among survivors. Reinforcement learning offers a route to data-driven treatment policies, but clinical adoption is limited by concerns that interpretability must be sacrificed for performance and that interactive (online) training is inherently superior to learning from static (offline) data. We benchmark these assumptions by comparing outcome performance and quantitative interpretability across offline and online reinforcement learning approaches for sepsis treatment. Methods: Using the gym-sepsis simulator derived from intensive care electronic health records, we trained offline policies with behavior cloning and conservative Q-learning on a fixed dataset of 2,105 simulated episodes (approximately 20,000 transitions) generated by a guideline-inspired heuristic policy. We also trained three online methods (double deep Q-network with attention, double deep Q-network with residual connections, and soft actor-critic) with simulator interaction, and included a vanilla deep Q-network as a baseline trained via interaction. Policies were evaluated on 500 newly sampled episodes per method, reporting overall survival and survival stratified by initial Sequential Organ Failure Assessment severity. Interpretability was quantified via linearly estimated gradients feature-saliency analysis. Results: Survival rates were tightly clustered (94.0%–95.4%) across learned policies, and the random baseline also achieved high survival, indicating limited outcome sensitivity within the simulator. For high-severity episodes (Sequential Organ Failure Assessment score ≥ 11), the attention-based online method reached 90.5% survival, compared with 88.5%–88.6% for conservative Q-learning and behavior cloning, while the vanilla deep Q-network underperformed (84.3%). Interpretability differed substantially: conservative Q-learning produced the strongest saliency signals (maximum saliency 40.06, emphasizing clinically relevant hemodynamic features such as systolic blood pressure), online methods showed intermediate magnitudes (maximum 1.17–3.57), behavior cloning was lower (0.78), and the vanilla deep Q-network was negligible (0.069). Conclusions: In this sepsis treatment simulator, online methods provided at most marginal survival improvements, while conservative offline learning achieved comparable outcomes with markedly stronger and more clinically coherent feature-importance signals and without requiring exploratory interaction during training. These findings support prioritizing conservative offline approaches when safety and explainability are central, while emphasizing that generalization beyond simulation requires prospective validation.