Detection of stegware using variational autoencoders and neural representation of noise

Abstract The increasing sophistication of cyber threats has led to stegware—malicious payloads hidden within digital images using steganographic techniques—posing serious detection challenges. This study introduces a variational autoencoder (VAE)-based framework for stegware detection that identifies hidden malware artifacts through residual noise analysis. A synthetic dataset of 20 000 grayscale images was generated by embedding three malware families (AsyncRAT, Lumma-Stealer, and Knot-Wed) into BOSSBase images using least significant bit encoding. Unlike conventional classifiers, the proposed VAE learns the latent distribution of clean images and isolates anomalous perturbations caused by malware embedding. Experimental results show that the model achieves 98.24% accuracy and an F1-score of 98.24%, outperforming convolutional neural networks and Autoencoder baselines under identical conditions. The findings demonstrate that latent-space deviation and reconstruction loss in VAEs can serve as reliable indicators of stegware presence, offering a lightweight, interpretable, and scalable solution for malware forensics.