AaSP: Aliasing-Aware Self-Supervised Pre-Training for Audio Spectrogram Transformers

Transformer-based audio self-supervised learning (SSL) models often operate on spectrogram-based time-frequency representations and apply vision-style Transformer architectures (e.g., ViT-style) together with masked modeling objectives (e.g., MAE/MIM-style). Typically, they use convolutional patchification with temporal downsampling. This lowers the effective Nyquist frequency and introduces aliasing, while naïve low-pass filtering removes task-relevant high-frequency cues. In this study, we present AaSP, an aliasing-aware self-supervised pre-training framework for audio spectrogram transformers. AaSP combines an aliasing-aware patch representation, teacher-student masked modeling, a cross-attention predictor, and multi-mask contrastive regularization to learn representations that can better integrate features derived from alias-prone modulation bands while remaining more stable across masked views. Its patch-embedding component, Aliasing-aware Patch Embedding (AaPE), augments standard patch tokens with features derived from alias-prone modulation bands using a band-limited complex sinusoidal kernel with a two-sided exponential window. The kernel's frequency and decay parameters are estimated from the input, enabling adaptive subband analysis whose outputs are fused with the standard patch tokens. These components encourage consistency across masked views, which appears important for realizing the empirical benefit of the aliasing-aware patch representation. We pre-train on AudioSet and evaluate the learned representations via fine-tuning and linear evaluation on downstream benchmarks covering general acoustic/environmental, speech-related, and music-related audio recognition tasks. Under fine-tuning, the full AaSP framework achieves state-of-the-art results on AS-20K, ESC-50, and NSynth among the compared self-supervised baselines, while remaining competitive on the other benchmarks. Linear evaluation shows a broadly consistent trend, with clear improvements on several benchmarks (e.g., US8K and NSynth). Overall, these results suggest that the full AaSP framework can learn representations that are more stable under aliasing-sensitive temporal perturbations and competitive for downstream transfer.