Bring KANs to Tracker: A Nonlinear Fusion Strategy Over Frequency Decoupling for Temporal Hyperspectral Object Tracking

Hyperspectral (HS) cameras have great potential in extracting spectral, textual, and temporal information from objects. Many existing works leverage HS data for object tracking, as it provides unique spectral features that can help address challenges like background clutter or camouflage. However, most of these methods overlook the rich temporal information available in video sequences, and many spectral-visual fusion approaches fail to extract contextual information from a global perspective, causing an insufficient understanding of an entire object by the model. To address the above issues, a spectral-temporal tracking Transformer based on a frequency-domain fusion strategy (S3T-FFS) is proposed. Firstly, a spectral-temporal token is proposed to capture an object’s spectral information that remains unchanged in video clips, providing additional tracking cues. Secondly, to extract spectral semantic information, we propose a frequency-domain fusion strategy (FFS), including a frequency attention network (FAN) and a Kolmogorov–Arnold Network-based convolutional unit (CuKAN), to provide spectral information for the tracking model. Specifically, FAN is designed for the simultaneous extraction and fusion of spectral features. This synchronous modeling approach decouples low-frequency and high-frequency information, adjusting their balance through frequency-domain prior knowledge and self-adaptive weights. This allows the fusion network to focus more on global information. To further extract global patterns from the low-frequency features and improve the network’s interpretability, we introduce CuKAN to extract nonlinear relationships within the decoupled frequency components, and its learnable activation function helps the model learn global patterns, thus avoiding the local overfitting in convolutional neural networks. Extensive experiments on multiple large-scale datasets illustrate the effectiveness of our proposed methods.