GXMalDetect: A Hybrid GA–XGBoost Architecture for Malware Detection Using Static Image Features

Malware is increasing in both volume and sophistication, and evasion strategies such as polymorphism, packing, and obfuscation reduce the effectiveness of signature-based detection. To enable accurate yet deployable malware classification, this paper proposes a lightweight hybrid framework that integrates Genetic Algorithm (GA)-based feature selection with an Extreme Gradient Boosting (XGBoost) classifier, aiming to reduce computational overhead while improving interpretability. Experiments are conducted on the Malimg benchmark, which contains 9,339 grayscale malware images from 25 families. We first extract a compact set of handcrafted static image features capturing statistical descriptors, texture/GLCM relationships, and morphological (shape) properties. The dataset is split using stratified 80:20 sampling to preserve class distributions, and GA optimization is performed only on the training split to avoid data leakage. Using cross-validated performance with a sparsity-aware fitness objective, GA converges to nine informative features spanning statistical (variance, energy, entropy, skewness, kurtosis), texture/GLCM (dissimilarity, homogeneity, correlation), and shape (compactness) descriptors. Trained on this subset, XGBoost achieves 99.30% test accuracy, 99.32% precision, 99.30% recall, 99.30% F1-score, and an MCC of 0.9918. ROC and precision– recall analyses further indicate near-perfect class separability. These results show that carefully selected handcrafted features coupled with gradient boosting can match or exceed heavier deep learning pipelines, providing a practical, fast, and interpretable solution for real-world malware classification.