TILES: Transformer-Based Incremental Learning for Expanding Segmenter

Incrementally learning new semantic concepts while retaining old knowledge is crucial for several real-world applications. Although the impact of backbone size and architectural choices has been extensively studied in non-incremental computer vision tasks for efficiency concerns, class-incremental semantic segmentation models have predominantly relied on large backbones, without providing alternatives with smaller models. Moreover, existing works use non-comparable backbones, which makes it difficult to isolate the impact of the incremental strategy versus the architectural improvements. In this work, we conduct a fair study across existing class-incremental semantic segmentation approaches, disentangling the incremental mechanism gains from the underlying architectural influence. Besides, we introduce TILES (Transformer-based Incremental Learning for Expanding Segmenter), a novel method that leverages the efficiency of small ViT backbones to provide an alternative solution under strict memory constraints. TILES incrementally expands the architecture, enabling learning new concepts while retaining previously acquired knowledge within a limited memory footprint. To address the challenge of background semantic shift, we employ adaptive losses specific to the incremental branches, balancing old and new information. Moreover, we exploit the confidence gained by the model through training data at each increment to propose an efficient branch merging strategy. TILES represents a compelling first alternative for scenarios with stringent memory and computational limitations. Extensive evaluations on challenging benchmarks demonstrate that using smaller backbones result in a drastic performance drop for the current state-of-the-art method, while TILES succeeds to maintain interesting results proving that it is the very suitable for such scenarios.