S2CaT: Spatial–Spectral-Based CNN and Transformer Network for Hyperspectral Image Classification

Recent advancements in the field of hyperspectral image (HSI) analysis have highlighted the potential of hybrid architectures that integrate convolutional neural networks (CNNs) with Transformer modules. These architectures are capable of effectively modeling global dependencies while preserving local feature representations. However, these methods fail to fully exploit the joint spatial–spectral features of HSI and instead extract only local or global features separately through CNNs or Transformers. To address these limitations, we propose a novel spatial–spectral-based CNN and Transformer network named S2CaT. In the early stage of the network, dedicated Spatial–Spectral Extractor (SSE) are employed to capture joint spatial and spectral features. Specifically, the Grouped Multi-Scale Spectral Extraction (GMSpe) is responsible for band-wise spectral processing, while the Cascade Spatial Extraction (CaSpa) extracts continuous spatial representations by aggregating multi-level spatial information. Subsequently, we introduce the CaT Fusion module to capture both local and global spectral contextual dependencies. This module comprises two key components: the Dilated Scale CNN (DS CNN) and the Domain Enhanced Transformer (DE Transformer). DS CNN leverages dilated convolutions with varying receptive fields to extract local features and enhance sensitivity to fine-grained textures. The DE Transformer replaces traditional linear projections with channel reorganization for generating the QKV matrices. Channel and spatial attention mechanisms are applied to the Q matrix to enrich feature diversity, while downsampling is introduced in the K and V matrices to suppress noise bands, facilitating robust feature aggregation and cross-domain interaction. We conducted experiments on the Indian Pines, Pavia University, Houston2013, and WHU-Hi-LongKou datasets using 5%, 0.7%, 5%, and 0.1% of the labeled data for training, respectively. The proposed method achieved improvements in average accuracy of at least 0.52%, 1.48%, 0.96%, and 1.07% over several recent state-of-the-art (SOTA) approaches on the corresponding datasets. We have released the full source code to ensure reproducibility. It is available at: https://github.com/ZH3nGsQ/S2CaT.