AI-Driven Engineering Techniques for Heterogeneous Medical Data:Facilitating Reuse in Clinical and Research Applications

The advances in healthcare technology has led to substantial rise in the volume and variety of medical data collected in the hospitals. This collection of data has become an integral part of routine clinical care, leading to availability of large amount of patient data available for the clinicians for decision-making. In the context of advancing medical research and decision support systems, these extensive medical datasets serve as a treasure trove of valuable, hidden insights, enabling a wide range of analytical explorations. These datasets act as an invaluable repository for development of Artificial Intelligence (AI) Models for Prediction/Detection/Risk Stratification in clinical workflows, work as abasis for statistical assessments for clinical studies, and can also act as patient cohorts for In Silico Clinical Trials (ISCTs). Despite these benefits, significant challenges arise in leveraging the reuse of medical data beyond routine care, largely due to heterogeneity — the inherent variability across different datatypes such as medical images, physiological signals, patient records, and computationally derived structures like segmented geometries. The reuse of heterogeneous medical data for different studies presents significant barriers include availability of limited data, incomplete data and imbalance in the data. These barriers pose a threat to the reliability of AI-driven prediction models and also have a negative impact on the statistical evaluations and clinical research. Though there is availability of datasets, but these datasets are mostly small in size, comprising of few hundred patients. This makes it difficult to reuse them to develop AI models or utilize them as cohorts for ISCTs. These limited datasets are inadequate for conducting any of the aforementioned studies. Other than this, during longitudinal data collections, it is often seen that there are quite some measurements missing for patients due to several reasons such as missing appointments, technical issue with the measurement devices or dependency on third party data collection centers etc. Lastly, it is extremely hard to obtain a dataset that has equal number of samples for two or more different classes. The use of these kinds of imbalanced datasets would lead to biased AI models, insignificant statistical analyses and misrepresented ISCTs. In order to address these challenges and create an efficient workflow for reuse of the medical data for different domains of applicability, this thesis presents AI-based data engineering methods for heterogeneous medical data. These data engineering methods comprise of the data augmentation, data imputation and minority class balancing techniques. This dissertation focuses on outlining a generalized pathway or rationale for the appropriate utilization of these engineering techniques with respect to the data source and its nature. Furthermore, this research work also forays into AI-based applications to demonstrate the applicability of the techniques for definite applications in clinical care. The studies presented in the thesis showcase the research on multitude of medical data in order to present a panoramic view of the AI-based data engineering solutions for multifarious sources of data in hospitals/clinical care. This research work has been pursued to obtain an inherent understanding of the field-sensitive issues in re-use of medical data for advanced healthcare technology development. This work mainly aims to harness the AI techniques for obtaining substantial, complete and balanced datasets, fulfilling the fore-most requisite for AI/statistical/clinical studies. This thesis is organized in parts with the objective to address the aforementioned challenges. Chapter1 introduces the the central obstacles to the secondary use of medical data, establishing a conceptual basis for AI-driven data engineering in healthcare. After introducing the key impediments, the thesis organizes the solutions in three structural units namely data augmentation, data imputation and data balancing. The first part i.e., data augmentation comprises of two chapters: Chapter2 and Chapter 3. Part I explores methodological innovations to enhance both the variability and fidelity of medical data through augmentation strategies. In this part, Chapter 2 presents a widely-applicable generative AI framework for synthesizing anatomically diverse CT images, demonstrating its utility in augmenting imaging datasets for ISCTs. It further introduces a multi-phase verification and validation protocol to ensure the credibility and clinical relevance of the resulting virtual cohorts. Chapter 3 shifts the focus from raw medical imaging to the augmentation of derived anatomical data, specifically the 3D geometries extracted through segmentation. It introduces a hybrid data augmentation strategy that integrates statistical modeling with generative AI to enhance the diversity and realism of these geometric datasets. This approach enables the construction of virtual cohorts representative of specific sub-populations, thereby supporting targeted evaluation of medical devices and therapeutics within ISCT frameworks. Second part i.e., data imputation presents an extensive analytical chapter for maintaining the integrity of electronic health records (EHRs). In this part, Chapter 4 addresses the issue of missing data in electronic health records (EHRs) and presents a comparative assessment of imputation methods. Based on dataset characteristics, missingness levels, and application needs, an imputation selection pathway is proposed. This provides guidance for choosing suitable approaches in clinical studies and AI model development. Last part, i.e., data balancing comprises of three chapters spanning over different data balancing techniques for EHRs and physiological signals. Chapter5 provides an analytical evaluation of methods for balancing patient characteristics in EHRs. It compares traditional resampling techniques with generative AI approaches for creating synthetic data. The chapter concludes with an assessment of these methods to support informed strategies for mitigating bias in EHR datasets. The data balancing techniques are also examined in the context of physiological signals such as electrocardiograms (ECGs) and electroencephalograms (EEGs). Chapters 6 and 7 investigate different strategies for addressing class imbalance in physiological signal datasets. Chapter 6 ex-amines the impact of resampling technique on imbalanced electrocardiogram(ECG) datasets with differing degrees of imbalance, while Chapter 7 explores the use of customized loss functions to mitigate bias in slightly imbalanced electroencephalogram (EEG) datasets. Together, these chapters provide a comparative analysis of balancing approaches for AI-based prediction models, highlighting their effectiveness across varying levels of data imbalance in biomedical applications. Finally, Chapter 8 presents an overview of the preceding chapters, outlines future research directions, and provides concluding insights emphasizing the significance of AI-based data preprocessing in ensuring data quality for secondary medical data applications.