PanoEarth: Scaling Open-Vocabulary Panoptic Segmentation in Remote Sensing Images

Existing open-world remote sensing foundation models rely on text-to-image alignment for open vocabulary semantic segmentation or object detection. However, semantic segmentation lacks instance-level delineation, while object detection fails to provide dense prediction for uncountable geographic features. Current approaches lack a unified solution for comprehensive panoptic understanding that jointly addresses semantic segmentation, instance segmentation, and object detection across all geographic elements. To bridge this gap, we propose PanoEarth, the first open-vocabulary panoptic segmentation framework for remote sensing images, which integrates these tasks within a unified architecture. Our framework incorporates a scalable training strategy to compensate missing background semantic information in instance segmentation data, enabling dense panoptic perception. Through the introduction of a domainspecific text embedding classifier to guide fine-tuning and establish precise mask query-to-category alignment, PanoEarth achieves competitive zero-shot open-vocabulary performance. We curate a large-scale panoptic dataset comprising 150k images spanning 111 geographic categories and millions of instances across landcover types, infrastructure elements, and fine-grained object subtypes. In zero-shot experiments, PanoEarth surpasses task-specific state-of-the-art models across multiple datasets and tasks. The joint co-training version enables unified extraction of 111 geographic categories, enabling hierarchical analysis from landcover mapping to fine-grained instance recognition. The source code is available at: https://github.com/xypu98/PanoEarth.