The Selection Principle in Transformer Attention: Fundamental Capacity Bounds from Information Theory

Technical Abstract: We derive an information-theoretic upper bound on the distributional capacity of transformer attention mechanisms. The Selection Principle states that the maximum meaningful precision achievable by any single parameter is bounded by log₂(V) bits, where V is the vocabulary size. Combined with the observation that single-head attention has d² effective degrees of freedom (from the bilinear interaction W_Q W_K⊤), we obtain the Capacity Theorem: C(d) ≤ d² × log₂(V) bits per token per layer. Accessible Explanation: Large language models use attention to decide which words influence each other. This paper asks: how much information can attention actually carry? We derive a mathematical limit, like the maximum data rate of a cable. The key finding: attention capacity is bounded by d² × log₂(V) bits per word per layer. Models with multiple attention heads trade raw capacity for specialisation — explaining why GPT-4 uses 128 heads rather than one giant head. Scientific Significance: This is the first information-theoretic derivation of transformer capacity from first principles. The bound explains empirical scaling laws (Kaplan 2020, Hoffmann 2022) and predicts architectural choices in production LLMs. The Selection Principle completes the "Convergence Arch" with the Russian Dolls Limit — together establishing that both classical retrieval and neural attention converge on mathematically optimal architectures.