An Investigation on External-Internal Drift in Faithfulness Evaluation in Hyperbolic Prototype-based Classifiers

The paper aims to study faithfulness evaluation in hyperbolic prototype-based vision models throughthe interaction between external perturbation on pixels, and internal perturbation that is geometricallyconsistent. Although insertion/ deletion based tests are widely used to assess the hypothesis whethersalient pixels are substantially capable of affecting model output, they do not by themselves provideevidence whether the induced latent evolution is semantically consistent with the internal geometry ofthe model. The research addresses the limitation by introducing a two-way framework that comparesthe external perturbation trajectories with internal tangent-space and geodesic trajectories under acommon semantic reference and a common intrinsic progression scale. With this method, a drift measureis introduced, which quantifies the difference between externally effective relevance and internallyvalid semantic change. Empirically we observed that pixel-based perturbations do not alter genuinefaithfulness signal, but the signal is only partially consistent with the semantic structure of the model’sinternal structure. The observed discordance can be explained by the perturbation-side effects and thehyperbolic operating regime, especially near the boundary, where geometrical amplification increasesthe path-dependent variation. We therefore, argue that instead of solely depending on traditionalperturbation-based metrics of faithfulness, drift can be considered as a complement how externalrelevance can align or fail to align, and thus providing a deeper insight into the relationship betweenexternal relevance and internal semantic consistency.