Machine Learning-Enhanced Malware Obfuscation and Innovative Defense Strategies

Read full text
Authors: Preet Kanwal, Tejas Kumar, Sanjay Sunil, Srimanish Chandrasekaran, Sajal Jaiswal, Prasad B. Honnavalli, S S Iyengar
Cited by: 1
References: 64

Abstract

In the evolving landscape of sustainable digital technologies, safeguarding cyber-ecosystems has become a critical priority. Traditional machine learning-based malware detection systems are increasingly ineffective against sophisticated adversarial techniques that exploit system-level vulnerabilities and bypass standard security protocols. This paper introduces a novel machine learning-driven malware obfuscation methodology and proposes a comprehensive defense strategy to counteract such threats. By leveraging Generative Adversarial Networks (GANs), combined with advanced encryption techniques and structural modifications of Portable Executable (PE) files, we generate highly obfuscated malware capable of evading conventional detection mechanisms. These malicious payloads are engineered for stealth, enabling seamless reconstruction and execution on target systems with minimal user interaction, while remaining undetected by over 90% of existing antivirus and endpoint security solutions. The residual detection rate is largely attributable to encryption heuristics rather than behavioral indicators. We also underscore the often-overlooked role of executable author privileges in facilitating unauthorized access through privilege-aware execution flows. To mitigate these risks, we present an innovative defense framework that enhances endpoint protection and strengthens the resilience of digital infrastructure against advanced obfuscation-based threats. Our approach is validated through the development of a custom image-based malware dataset, enabling both visual and behavioral analysis of obfuscated samples, and offering a new paradigm for malware research and cybersecurity defenses.

Fields of Study & Subjects

Advanced Malware Detection Techniques Adversarial Robustness in Machine Learning Security and Verification in Computing

Publication details

Journal / source
IEEE Access
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Volume
14
Pages
12605–12627
Publication date
1 January 2026