Weakly Supervised Classification with Pre-Trained Models: A Robust Fine-Tuning Approach

Weakly supervised classification (WSC) is a popular machine learning paradigm that aims to train a classifier using incomplete, inexact, or inaccurate supervision. Recently, it has become common practice to use a general-purpose, large, pre-trained model as a foundation model that is fine-tuned to solve complex, challenging downstream classification problems. However, collecting fully supervised downstream data can be costly in certain domains. Thus, it makes sense to apply the WSC paradigm to the fine-tuning scenario. In this paper, we attempt to fine-tune a pre-trained vision transformer using the WSC approach. Our experiments show that naive use of existing WSC losses degrades performance due to severe overfitting exacerbation and feature degeneration problems. To address these problems, we propose a novel robust fine-tuning approach using dual classification heads that are trained synergistically by alternately distilling reliable supervision and performing efficient model fine-tuning. Theoretically, we prove the consistency and convergence rate for the proposed risk estimator. Empirically, extensive experimental results on diverse benchmark datasets validate the effectiveness of our proposed approach against state-of-the-art approaches.