ProtoFormer: Transformer With Learnable Prototype Tokens for Pansharpening

Transformers have achieved remarkable success in satellite pansharpening by effectively modeling non-local similarity. Self-attention focuses on modeling the global dependency among all token pairs, typically pixel pairs for pansharpening. Representative tokens with rich information are often ignored. This leads to redundant computation, especially for similar tokens. Instead, non-similar regions share the weights to aggregate features in a Feed-Forward Network (FFN), leading to suboptimal performance. To address these limitations, we propose ProtoFormer, a novel transformer architecture that introduces learnable prototype tokens to capture and leverage representative image structures, featuring two key contributions: 1) Shared Prototype Attention (SPA): it exploits the high redundancy and symmetry of attention maps by computing a single shared attention map between image and prototype tokens, enabling bidirectional exchange of global self-similarity information while reducing self-attention complexity from quadratic to linear; 2) Prototype-guided Content-Aadaptive FFN (PCAFN): it addresses the limitation of standard FFNs that use shared weights for all positions by leveraging prototype tokens to partition image tokens into similar-content regions, then applying region-specific content-adaptive convolution and a soft mask gate, thereby enhancing spatially adaptive feature aggregation and preserving fine edges. Extensive experiments demonstrate that the proposed method outperforms state-of-the-art fusion methods through both qualitative and quantitative assessments on multiple test sets. Our code will be released upon acceptance.