Softpick: No Attention Sink, No Massive Activations with Rectified Softmax

We introduce softpick, a rectified, not sum-toone, drop-in replacement for softmax in transformer attention mechanisms that eliminates attention sink and massive activations.Our experiments with 340M and 1.8B parameter models demonstrate that softpick achieves 0% sink rate consistently.The softpick transformers produce hidden states with significantly lower kurtosis and creates sparse attention maps.Quantized models using softpick outperform softmax on standard benchmarks, with a particularly pronounced advantage at lower bit precisions.Our analysis and discussion shows how softpick has the potential to open new possibilities for quantization, low-precision training, sparsity optimization, pruning, and interpretability.Our code: https://github.com/ zaydzuhri/softpick-attention. * Equal contributionSink Rate = 63.41%Sink Rate = 0.00%