Simulating Hard Attention Using Soft Attention
Abstract
Abstract We study conditions under which transformers using soft attention can simulate hard attention, that is, effectively focus all attention on a subset of positions. First, we examine several subclasses of languages recognized by hard-attention transformers, which can be defined in variants of linear temporal logic. We demonstrate how soft-attention transformers can compute formulas of these logics using unbounded positional embeddings or temperature scaling. Second, we demonstrate how temperature scaling allows softmax transformers to simulate general hard-attention transformers, using a temperature that depends on the minimum gap between the maximum attention scores and other attention scores.
Fields of Study & Subjects
Ferroelectric and Negative Capacitance Devices
Advanced Memory and Neural Computing
Parallel Computing and Optimization Techniques
Publication details
- Journal / source
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Volume
- 14
- Pages
- 147–166
- Publication date
- 16 January 2026
- Language
- EN