ViM-Q: Energy Efficient Algorithm-Hardware Co-Design for Dynamically Quantized Vision Mamba Models

State-space models (SSMs), such as Mamba, provide an efficient alternative to Transformers for vision tasks by replacing their quadratic-cost self-attention with linear complexity state update. However, efficiently deploying Vision Mamba (ViM) models on FPGA platforms is challenging, as the latency is dominated by two key components: linear layers and the selective SSM. For the linear layers, highly dynamic activation outliers across tokens render conventional static quantization techniques ineffective. Meanwhile, while the associative scan algorithm is effective in accelerating SSM on GPUs, its data access pattern is fundamentally mismatched with FPGA architectures when mapping the model's inherently sequential recurrence, creating a critical dataflow bottleneck.