A survey on design choices for self-supervised learning in computer vision

Abstract Self-supervised learning is a successful strategy to overcome data scarcity and improve robustness in computer vision by adding a pretext task that can exploit inherent data relationships as supervision signals during pretraining. However, the combination of pretraining and downstream training renders model design more complex, as additional design choices are required. This paper analyses the effects of such design choices specific to self-supervised learning on model performance and robustness. How does the pretext task influence the downstream task and how to design an ideal and generalizable pretext task? Which properties of the pretraining dataset are favorable and how similar should the pretext and downstream dataset ideally be? To address these questions, a comprehensive survey has been conducted, encompassing the results of diverse models and publications with different design choices. The results demonstrate the advantages of in-domain pretraining and the importance of aligning all design choices in order to ensure optimal results. Furthermore, the characteristic differences between predictive, contrastive and generative self-supervised learning and the design choices which are crucial for each of these learning paradigms are analyzed in detail.