RECALL+: Adversarial web-based replay for continual learning in semantic segmentation

Catastrophic forgetting of previous knowledge is a critical issue in continual learning, typically handled through various regularization strategies. However, existing methods struggle especially when several incremental steps are performed. In this paper, we extend our previous approach (RECALL) and tackle forgetting by exploiting unsupervised web-crawled data to retrieve examples of old classes from online databases. In contrast to the original methodology, which did not incorporate an assessment of web-based data, the present work proposes two advanced techniques: an adversarial approach and an adaptive thresholding strategy. These methods are utilized to meticulously choose samples from web data that exhibit strong statistical congruence with the no longer available training data. Furthermore, we improved the pseudo-labeling scheme to achieve a more accurate labeling of web data that also considers classes being learned in the current step. Experimental results show that this enhanced approach achieves remarkable results, particularly when the incremental scenario spans multiple steps. • The paper tackles catastrophic forgetting in continual semantic segmentation with a web rehearsal-based approach. • It introduces a novel strategy to select the web images that will be used for replay, featuring an adversarial learning scheme together with an adaptive thresholding approach applied on semantic pseudo-labeling. • It introduces a novel self-inpainting approach that also inpaints predictions for the new classes being learned. • A large experimental evaluation, including the challenging ADE20k dataset, shows the effectiveness of the approach.