Dyn-SSM: Towards the Efficient Long Sequence Learning via Bio-interpretable Dynamics in Spiking State Space Models

Spiking neural networks (SNNs) provide a lowpower, energy-efficient solution by utilizing the spike-based and sparse nature of biological systems. Since the advent of Transformers, SNNs have struggled to compete with artificial networks on long sequential tasks, until the recent emergence of state space models (SSMs), which offer superior computational efficiency and modeling capability. However, despite the initial effectiveness of applying highly capable SSMs to SNNs for long sequence learning, existing SNN-based SSMs ignore the complex dynamics of biological spiking neurons, which are crucial for the functionality but challenging to simulate and exploit effectively in large networks. In addition, the membrane potential is determined by the past spiking history of the neuron, leading to reduced efficiency for sequence modeling in parallel computing scenarios. Therefore, we propose Dyn-SSM, which aims to explore and leverage biologically motivated and more interpretable dynamics in SNN-based SSMs for long sequence learning. Specifically, we propose to incorporate the reset-refractory mechanism in SNN-based SSMs to leverage the inherent temporal dimension for long sequence modeling. Then, to enable parallel processing, a novel parallel max-min boundary compression strategy (PMBC) that can be widely applied is proposed to accelerate the inference process. Finally, we hierarchically integrate the proposed neuron model to the original SSM block, and enhance the dynamics of Dyn-SSM by incorporating trainable thresholds and refractory magnitudes to balance accuracy and sparsity. Extensive experiments illustrate the effectiveness and robustness of Dyn-SSM, showing the potential of dynamic spiking neurons for balancing sparsity and precision in efficient long sequence learning. The code and Supp. Mat. are publicly available inhttps://github.com/zhyan567/Dyn-SSM.