Linear Attention as Bayesian Inference
Abstract
This paper shows that the recurrent update used in linear attention models is mathematically equivalent to Bayesian inference under a Dependent Dirichlet Process. The forgetting rate, the input-dependent gate, and the multi-head structure each receive a clean probabilistic interpretation. The result is exact and holds by construction, not as an approximation.
Fields of Study & Subjects
Topic Modeling
Gaussian Processes and Bayesian Inference
Neural Networks and Applications
Publication details
- Journal / source
- Zenodo (CERN European Organization for Nuclear Research)
- Publication date
- 12 March 2026
- Language
- EN