Universal Domain Adaptation Meets Multimodality Remote Sensing Data for Scene Classification

Universal domain adaptation is a practical technique for remote sensing scene classification, since the expensive collection of labels in remote sensing scenarios usually results in the relationship of the label sets between the source domain and the unlabeled target domain being unknown. The domain shift and the unknown category shift are two important challenges for the universal domain adaptation task. To reduce these shifts, we propose a Multi-modal Universal Domain Adaptation (MUniDA) method for remote sensing scene classification by using both the source images and their captions. MUniDA first aligns the source images and the textual captions with CLIP model to mitigate the gap of cross modalities. Two classifiers are trained following the visual encoder and text encoder of CLIP with source images and their captions, respectively. Thus the representation of the image can be much more discriminative for classification due to the rich information in the textual captions. Then a target fine-tuning method is developed with an orthogonal feature decomposition strategy and a pseudo-labeling strategy. The decomposition strategy is designed by decomposing the class prototype of source images with singular vector decomposition into the orthogonal base vectors of the known class and the unknown class, which are useful to distinguish the common data into target samples into common data and private data. Meanwhile, the pseudo-labeling strategy is proposed by assigning confident pseudo-labels to the common data with common labels while assigning an unknown label to the private data with different classifier predictions for fine-tuning. Experimental results demonstrate that our proposed method outperforms the state-of-the-art universal domain adaptation methods.