Frequency-Aware Visual Restoration and Cross-Stream Multimodal Attention for Action Classification in Degraded Videos

Video recognition of human activities is severely limited because downscaled video has lost the spatial and motion information required to identify human activity. Common methods used to restore lost information, like handcrafted feature extraction algorithms or pixel interpolation-based upscaling, fail to restore the essential visual elements that are lost forever when the original input is less than 20 × 20 pixels. Pixel interpolation alone will only recover the original degraded data but will usually introduce unwanted shimmering artifacts or false contour lines, which add to the difficulty of recognizing activity in an input video.In addition, convolutional and other conventional neural networks that are developed for use with full-resolution video see their performance drop dramatically due to their inability to extract meaningful discriminative features from semantically poor visual signals produced by very low-resolution videos.To address this limitation, we propose a dual-module framework designed to retrieve meaningful visual content from severely degraded input videos and improve recognition performance. The first module, the Multi-Scale Reconstruction Module (MSRM), retrieves visual data from degraded video using a frequency-structure adaptive approach. Each frame is decomposed into low-frequency and high-frequency bands using a discrete wavelet transform. Low-frequency components that contain global semantic information are reconstructed using the Enhanced Deep Super-Resolution (EDSR) network, while high-frequency textures and fine details are recovered using a Wavelet-based Super-Resolution Generative Adversarial Network (WSRGAN). This frequency-based decomposition leverages statistical similarity among sub-band signals, enabling stable reconstruction of global information through residual learning and visually accurate restoration of fine details through adversarial learning.The reconstructed RGB frames (spatial information) and their corresponding optical flow data (temporal information) are then passed to the Multi-Modal Fusion Module (MMFM), which employs a two-stream Vision Transformer (ViT) architecture for analysis. Bidirectional cross-attention enables complementary feature extraction between spatial and temporal modalities, while temporal consistency is further enhanced through evaluation using temporal PSNR (t-PSNR) and temporal SSIM (t-SSIM).Experimental results demonstrate that the proposed framework achieves an accuracy of 62.25% on HMDB51 at 14×14, 76.74% on UCF101 at 14×14, and an F1-score of 35.63% on TinyVIRAT, with a total computational cost of 45.3 GFLOPs. This is significantly lower than existing approaches such as SlowFast, which require 213.0 GFLOPs, while still producing more precise results with substantially reduced computational cost.