Language-Informed Hyperspectral Image Synthesis for Imbalanced-Small Sample Classification via Semi-Supervised Conditional Diffusion Model

Data augmentation effectively addresses the imbalanced-small sample data (ISSD) problem in hyperspectral image classification (HSIC). Although most methodologies extend features in the latent space, few leverage text-driven generation to create realistic and diverse samples for balancing limited annotated data. This paper proposes Txt2HSI-LDM(VAE), a novel language-informed hyperspectral image synthesis method to address the ISSD problem in HSIC. The proposed approach uses a denoising diffusion model, which iteratively removes Gaussian noise to generate hyperspectral samples conditioned on textual descriptions. Firstly, to address the high-dimensionality of hyperspectral data, a universal variational autoencoder (VAE) is designed to map the data into a low-dimensional latent space, which provides stable feature representation and significantly reduces the inference complexity of the diffusion model. Secondly, a semi-supervised diffusion model is designed to fully take advantage of unlabeled data. Random polygon spatial clipping (RPSC) and uncertainty estimation of latent feature (LF-UE) without parameters are used to simulate the varying degrees of mixing of training data. Thirdly, the VAE decodes HSI from latent space generated by the diffusion model with the language conditions as input, contributing to more realistic and diverse samples that match the textual descriptions. In our experiments, we fully evaluate synthetic samples’ effectiveness from the aspects of statistical characteristics and data distribution in 2D-PCA space. Additionally, the key component, visual–linguistic cross-attention, is visualized on the pixel level to prove that our proposed model can capture the spatial layout and geometry property of the generated hyperspectral images. Experiments carried out on two widely used HSI datasets demonstrate that the proposed Txt2HSI-LDM(VAE) outperforms the classical backbone models, state-of-the-art CNNs, semi-supervised methods, visual–linguistic feature fusion method, as well as Diffusion-model-based methods.