CoMasTRe+: Unleashing Disentangled Continual Segmentation With Mixture of Continual Adapters

Continual Semantic Segmentation (CSS) suffers from catastrophic forgetting, particularly challenging for traditional per-pixel methods. Our prior work, CoMasTRe (CVPR 2024), introduced a query-based approach leveraging objectness by disentangling CSS into objectness learning and class recognition stages. While effective, CoMasTRe exhibited performance limitations due to feature forgetting within its pixel decoder. This paper presents CoMasTRe+, an enhanced framework specifically designed to overcome this limitation. The core contribution is a novel plugin, the Mixture of Continual Adapters (MoCA), integrated into the pixel decoder. MoCA is a dynamic architecture that mitigates feature forgetting by learning task-specific expert adapters. Crucially, MoCA employs a task-aware routing strategy and a novel adaptive routing distillation objective, tailored for continual learning, to preserve specialized feature representations across sequential tasks. CoMasTRe+ further enhances the class decoder using MoCA for improved recognition and simplicity. We extensively evaluate CoMasTRe+ on PASCAL VOC and ADE20K for continual semantic and panoptic segmentation. Experiments demonstrate that CoMasTRe+ effectively addresses the identified feature forgetting issue, significantly outperforms the original CoMasTRe, and achieves state-of-the-art results compared to both per-pixel and query-based baselines.