Learning Robust and Transferable Representations from Self-Supervision

Learning to represent the physical world from data observations lies at the heart of modern artificial intelligence (AI). While early research largely relied on explicit human supervision, recent developments have increasingly embraced weaker forms of supervision to drive representation learning. Among these, leveraging co-occurring observations, such as multiple modalities or views of the same scene, has emerged as a particularly powerful paradigm, enabling progress across diverse domains. Yet, despite these empirical successes, a comprehensive theoretical understanding of self-supervised representation learning remains elusive, limiting the design of methods that learn representations robust to data distribution shifts and transferable to new problems. To address this gap, this thesis develops theoretical explanations for why seemingly trivial self-supervised objectives can yield representations that generalize well beyond their training data and objective. In particular, we ask whether, in the infinite-data limit, solutions that optimize self-supervised objectives converge to equivalent representations. For instance discrimination, we find that discriminating between individual observations recovers the underlying latents up to linear transformations, assuming these latents are clusters on a hyper-sphere. In the multi-modal setting, we demonstrate that contrasting pairs of multi-modal observations leads to representations that capture latents shared across modalities, and discard modality-specific ones. Building on these theoretical insights, we introduce principled changes to existing heuristic-based self-supervised algorithms. These changes aim to improve the robustness of learned representations under data distribution shifts and enhance their transferability to new problems. First, we introduce the Principal Masked Autoencoder (PMAE), an auto-encoder trained to reconstruct randomly masked principal components, and show how the PMAE leads to improved downstream performance while substantially reducing hyperparameter tuning costs. Second, we propose a latent variable model inspired by contrastive learning. We find that the evidence lower bound (ELBO) derived from this model offers a principled learning objective that preserves semantic structure in the latent space while retaining information necessary for fine-grained tasks like generation. By investigating self-supervised learning, this thesis aims to clarify why and how the intrinsic structure of data can be harnessed to learn robust and transferable representations. Through identifiability results and the development of new self-supervised learning methods, it contributes to the broader goal of designing learning systems that require minimal human supervision yet generalize effectively across real-world datasets and tasks beyond language modeling.