Adaptive ensemble of multivariate filters with Harris Hawks optimization for feature selection in medical diagnosis

Feature selection (FS) constitutes an indispensable process in medical data analysis, pivotal for mitigating the curse of dimensionality while concurrently augmenting classification performance and fostering model interpretability. This article introduces a novel hybrid framework, the Adaptive Ensemble of Multivariate Filters (AE-MVF), which synergistically integrates Multivariate Mutual Information (MMI) with extreme gradient boosting (XGBoost)-based feature importance scores within a dynamically weighted ensemble architecture. The core of our contribution is the application of the Harris Hawks Optimization (HHO) algorithm to adaptively determine optimal filter weights and feature subset cardinality, a process explicitly designed to navigate the trade-off between maximizing feature relevance and minimizing inter-feature redundancy. The efficacy and robustness of AE-MVF were rigorously evaluated on a diverse suite of 22 benchmark medical datasets, spanning various dimensionalities, sample sizes, and class distributions. Empirical results demonstrate that AE-MVF yields statistically significant improvements over contemporary filter- and wrapper-based FS methodologies, establishing a new performance benchmark in both classification accuracy and feature subset parsimony. Notably, on high-dimensional, low-sample-size (HDLSS) datasets such as ‘Leukemia’ and ‘Colon’, the framework achieved substantial dimensionality reduction while preserving or enhancing predictive accuracy. In lower-dimensional contexts, AE-MVF delivered competitive performance with highly parsimonious feature sets, often comprising just 3–5 variables. These findings underscore the scalability and generalizability of the AE-MVF framework, positioning it as a potent tool for developing interpretable and computationally efficient models in real-world medical diagnostics.