Hybrid CNN–Transformer Framework for Behavior-Based Ransomware Detection Using System Call Traces

The problem of ransomware has become one of the most dangerous in terms of cybersecurity as it uses system-wide functions to bypass the legacy forms of signature protection. In order to overcome this, the current paper suggests an efficient behavior-based ransomware detection system based on system call trace and a hybrid deep learning system. The presented model combines the use of a Convolutional Neural Network (CNN) to obtain learning of fine-grained local behavioral patterns and a Transformer encoder to learn long-term execution dependencies with the help of self-attention modules. This bifurcated design presents the ability to model localized and global ransomware behaviors in detail in real-time. There are extensive experiments on a publicly available Kaggle ransomware detection dataset with stringent preprocessing and evaluation guidelines. The given method obtains a high detection rate of 98 %, and all three metrics of precision, recall and F1 scores are balanced, which proves the high level of generalization and high resistance to complex types of ransomware. The superiority of the proposed hybrid model is confirmed by comparison with CNN, ResNet, and Transformer baselines. The findings show the usefulness of convolutional and attention-based learning, when mixed to ensure viable and valid detection of ransomware in contemporary computing platforms.