Hybrid Learning Paradigms: Bridging Supervised and Unsupervised Methods for Semi-Supervised Deep Learning

The ever-increasing size of unlabelled data and the expensive cost for manual annotation have driven research on semi-supervised learning (SSL), which aims to take full advantage of labelled and unlabeled samples. There are various hybrid learning approaches that connect supervised and unsupervised counterparts to each other in the contemporary era, including self-supervised pretraining paradigms, consistency regularization methods, and pseudo-labeling practices. In this paper, we study three popular backbone architectures SimCLR, BYOL and MoCo to investigate their specific character is- tics, understand their theoretical roots and applications in both computer vision and natural language processing. The coming together of these two methods is a breakthrough in deep learning research, allowing researchers and developers to achieve almost-supervised performance with significantly less labels.