VALIANT: Prompt Instability for Active Learning in Black-Box Medical Imaging

The deployment of large, black-box foundation models for medical image classification is often hindered by the high cost of acquiring large, task-specific labeled datasets for fine- tuning. While active learning (AL) presents a promising solu- tion, many state-of-the-art AL methods are computationally expensive or require full access to internal model parame- ters. We present VALIANT (Visual Adaptation and Learning Integration for Active learNing Tasks), a new active learning framework designed to efficiently adapt black-box foundation models by overcoming these limitations. VALIANT intro- duces a lightweight Visual Prompt Decoder (VIPD), trained via unsupervised Zero-Order Optimization (ZOO), to gener- ate task-specific visual prompts without internal model access. Our core contribution is a perturbation-based ranking strat- egy that leverages this VIPD to formulate a computationally efficient, gradient-aware informativeness metric. This metric, which we term prompt instability, identifies the most impactful samples for the labeling budget. VALIANT further enhances this process by incorporating anatomical information from unsupervised segmentation maps to generate more discrimina- tive visual prompts. Extensive evaluations on multiple medical datasets demonstrate VALIANT’s superior performance and significant reduction in labeling costs compared to a range of existing active learning techniques, positioning it as a scalable and practical solution for medical image analysis.