Fourier–Transformer Mixer Network for Efficient Video Scene Graph Prediction

In video scene graph prediction, the aim is to capture structured object interactions that occur over time in dynamic visual content. While recent spatiotemporal attention-based models have improved performance, they often suffer from high computational costs and limited structural consistency across long sequences. Therefore, we developed a Fourier transformer mixer network (FTM-Net), a modular, frequency-aware architecture that integrates spatial and temporal modeling via spectral operations. It incorporates a resolution-invariant Fourier Mixer for global spatial encoding and a Fast Fourier Transform (FFT)-Net-based temporal encoder that efficiently represents long-range dependencies with less complexity. To improve structural integrity, we introduce a spectral consistency loss function that synchronizes high-frequency relational patterns between frames. Experiments conducted utilizing the Action Genome dataset demonstrate that FTM-Net surpasses previous methodologies in terms of both Recall@K and mean Recall@K while markedly decreasing parameter count and inference duration, providing an efficient, interpretable, and generalizable approach for structured video comprehension.