Filter Then Attend: Improving Attention-Based Time Series Forecasting with Spectral Filtering

Transformer-based models are at the forefront in long time-series forecasting (LTSF). Despite their powerful modeling capacity, they are hindered in this domain by a bias toward high-energy, low-frequency features in the data. Recent work has established that learnable frequency filters can strengthen deep forecasting models by enhancing their spectral utilization. These works choose to use multilayer perceptrons to process their filtered signals and thus do not address the issues found with transformer-based models. In this paper, we demonstrate that applying learnable frequency filters to embedded signals enhance the performance of transformer-based forecasters across a number of architectures with negligible increases in memory and compute. We also conduct synthetic experiments that demonstrate learnable filters enhance transformer-based models by amplifying primarily middle and high-frequency features in the data.