Explaining deep learning for ECG using time-localized clusters

OBJECTIVE: Deep learning has advanced electrocardiogram (ECG) analysis but remains difficult to interpret, limiting clinical adoption and electrophysiological insight. We propose a post-hoc explainability method for convolutional neural networks (CNNs) applied to ECG. METHODS: The method clusters the feature activations of the last three residual blocks of a 1D-ResNet, segmenting each ECG into a sequence of clusters and quantifying assignment entropy as a per-timestamp uncertainty. It is evaluated on PTB-XL super-class classification under 10-fold cross-validation and on CODE-15% age regression. RESULTS: Cluster proportions correlate with predicted labels and align with P/QRS/T/TP landmarks. A random forest trained on cluster proportions reproduces the CNN's predictions at $94.9 \pm 0.5\%$ and matches its accuracy and AUROC on the true labels (88.2 vs. $88.4\%$; 79.6 vs. $81.4\%$). Encoder uncertainty surfaces class-dependent representational stability not visible to Grad-CAM. CONCLUSION: Time-localized clusters recover physiologically aligned structure from CNN activations and faithfully summarize the encoder's discriminative information. SIGNIFICANCE: The method provides a post-hoc, architecture-light tool to audit CNN-based ECG models and surface label-quality issues.