Shuffle Counterfactuals with O(1)-in-H Geometric Masks Reveal Alignment Gaps

Hard top-k gating provides a simple route to sparse subnetworks, but its discrete nature makes it difficult to attribute why certain sparse masks perform well. We position this work as an analysis probe for sparse representations rather than a competitive sparsification method. We propose a diagnostic protocol-a shuffle counterfactual-to decompose performance gains in global hard masking. The protocol permutes the neuron-coordinate correspondence while keeping the coordinate multiset, mask family, and evaluation budget fixed, yielding an empirical no-alignment reference. To instantiate the protocol in a controlled and minimal setting, we introduce Geo-kWTA: a constant-parameter geometric family of global top-k masks optimized via gradient-free search on frozen features. Across sparsity sweeps on frozen ResNet-50 features, preserving alignment yields additional gains over the no-alignment reference, especially under strong sparsity, and the effect is depth-localized to late representation stages. We further show that geometry choices interact with alignment (e.g., Shell vs. center-distance), and that under distribution shift, robustness can be more alignment-sensitive than in-distribution behavior. Finally, we identify a three-way pattern across backbones: a positive alignment gap in ResNet-50 (layer 4), a negligible gap in CLS-pooled ViT-MAE, and a reversed (negative) gap in Augreg ViT-B/16 at strong sparsity-consistent with pretraining objective differences playing a role in shaping alignment-dependent behavior, though the observed variation across backbones may reflect a combination of factors beyond objective alone.