Learning from Multiple Data Modalities: Novel Model Architectures, Algorithms, and Applications to Healthcare Challenges

Anemia affects about one-third of the world’s population, yet current diagnostic methods often require invasive blood tests that limit comfort, accessibility, and scalability, particularly in low resource settings. This thesis aims to develop a noninvasive, accurate, portable, and low-cost anemia diagnosis solution using multimodal machine learning. We propose a system that integrates electronic health records (EHRs), photoplethysmography (PPG) signals, and conjunctival images to improve diagnostic performance while maintaining ease of use and accessibility. To achieve this, we address several main challenges: handling irregular and incomplete EHR data, ensuring robustness in multimodal transfer learning under data scarcity and missing modalities, and improving the reliability of multimodal transformers. We also explore training-free methods to enhance autonomous reasoning in multimodal large language models while reducing the need for extensive annotations and training. In addition, we have assembled a high-precision PPG system and prepared the necessary hardware to collect real-world datasets for validating our method. The contributions of this thesis are summarized as follows: we propose a multimodal anemia diagnosis method that integrates conjunctival images (image data), electronic health records (tabular data), and photoplethysmography (PPG) signals (time-series data). The proposed approach is accurate (high R² score in hemoglobin concentration prediction), portable, non-invasive, and cost-effective. The proposed methods are evaluated using data collected at University College London Hospital with NHS ethical approval. The proposed HemaFusion model, which integrates EHR data, PPG signals, and conjunctiva images, achieves an R² score of 0.775 ± 0.009 and an MSE of 0.173 ± 0.006 on hemoglobin concentration prediction, significantly outperforming other baselines, demonstrating that accurate and non-invasive anemia detection is feasible using our approach. This work lays the foundation for scalable, point-of-care anemia diagnostics that can improve patient care and broaden access to essential health screening, particularly in under-resourced regions. This thesis represents a milestone in practical application and effectively addresses pressing challenges in the field of anemia diagnosis.