Efficient Multi-Modal Learning under Resource Constraints

Leveraging information from multiple domains is crucial for applications such as health care and robotics, where integrating diverse data sources enables more accurate and robust decision making. However, in many real-world scenarios, access to all data types is limited due to incomplete inputs or constrained resources. This presents significant challenges for existing approaches, which typically assume the full availability of all modalities during both training and inference. To address these limitations, this thesis develops strategies for learning under missing or constrained multi-modal conditions. Specifically, we explore two complementary directions: 1) retrieval-based methods for bridging missing information across domains, and 2) adaptive learning for multi-modal analysis under resource constraints. For retrieval-based methods, we leverage a large-scale database to collect relevant cross-modal samples that serve as surrogates for the missing modality, thereby avoiding the need for potentially inaccurate synthesis. This strategy proves effective across various downstream tasks, providing strong approximations of the absent data. However, these methods can still struggle with highly heterogeneous modalities, such as between tabular and image data. To address this, we develop adaptive learning approaches that allow decision-making under limited resources. Given partially observed multi-modal data, our method can adapt to various patterns of missingness. Furthermore, when data is yet to be collected, we allow adaptive modality selection to balance between prediction accuracy and acquisition cost.