Get in Shape: Prevent Oversmoothing of CNN-Based Sentinel-2 Classification With Large Pretrained Segment Anything Model

Land Cover (LC) classification remains challenging, even with advanced machine learning (ML) techniques. Pixel-wise Convolutional Neural Networks (CNNs) are currently widely used for image classification. However, CNNs outputs often suffer from blurry boundaries and poorly segmented shapes. Hence, ad-hoc post-processing may be required to reduce over-smoothing. Conversely, it adds complexity to the classification process. Fortunately, generic foundation models like the Segment Anything Model (SAM), trained on large datasets, have become available. In this study, we propose a novel approach that combines outputs from locally trained CNN models and from the SAM model. We tested the approach on a mosaic of Sentinel-2 (S-2) images with a 10 m resolution, covering four neighboring first-level administrative units in Czechia. S-2 mosaic was classified according to the Land Use, Land-use Change, and Forestry (LULUCF) nomenclature using CNNs. Separately, this mosaic was segmented by SAM. The SAM output segments were assigned the dominant LC class from the CNN classification. The annotated segments were then overlaid on the original CNN classification to address over-smoothing. This approach resulted in an approximate two-percent improvement in overall accuracy (OA) and significantly enhancing the boundary delineation of LC objects in classification outputs, in terms of F1, precision, and recall. The reported two-percent improvement is meaningful because it was achieved on top of an already strong baseline (86.87 % overall accuracy), making further gains inherently difficult. LULUCF information is critical for monitoring compliance with the Paris Agreement. So, the ability to segment S-2 images into objects that align with LULUCF nomenclature could support future annual LULUCF reporting to the United Nations Framework Convention on Climate Change.