Enhancing Attention-Based Visual Processing With Noise-Boosted Activation Functions

Motivated by the principle of stochastic resonance, we investigate the noise-boosted activations within both channel attention mechanisms of convolutional networks and gated linear unit (GLU)-based feedforward networks (FFNs) of Vision Transformers (ViTs) under attention-based visual processing frameworks. Specifically, we replace conventional ReLU or ReLU-based GLU (ReGLU) activations with noise-boosted variants, which incorporate learnable noise scale parameters during training. Experiments on the CIFAR-10 and STL-10 image classifications, Kvasir-SEG medical image segmentation, and Cityscapes semantic segmentation show significant improvements over conventional baselines across diverse attention architectures. The learnable noise scale parameters in activations converge to non-zero values after training, demonstrating the existence of stochastic resonance in deep attention mechanisms. These results indicate that controlled noise injection can enhance information transfer efficiency of neural networks, and establish a coherent framework that connects the theoretical principle of stochastic resonance with its practical applicability in attention-based visual processing.