Spectral-Window Hybrid (SWH)

Scaling sequence modeling to extreme contexts requires balancing computational efficiency with rep-resentational expressivity. While Transformers provide precise retrieval via the attention mechanism,their quadratic O(T^2) complexity limits their application to long-horizon tasks. In this work, we proposethe Spectral-Window Hybrid (SWH), an architecture that decouples sequence modeling into twoparallel streams: a global branch utilizing the Convolution Theorem to model long-range decay dynamicsin O(T log T ) time, and a local branch employing sliding-window attention for token interactions withina bounded context. By aggregating these representations, SWH avoids the computational bottleneck ofglobal attention while retaining local precision. We demonstrate that SWH matches the perplexity ofstandard Transformers on short contexts while enabling efficient linear scaling to extended sequences.