Make Attention Simpler with Logit Mixing Attention (LMA)

Logit Mixing Attention (LMA) is an independent research exploration of an alternative attention mechanism designed to study how information mixing at the logit level influences learning behavior, entropy dynamics, and computational structure in neural sequence models. This work introduces LMA as a mechanism where query states are augmented by factorized information derived from keys and/or values, enabling competition and interaction in the model space dimension, rather than relying solely on token-to-token attention competition. Two primary variants are examined: LMA with K-Factor Mode, where keys act as scaling and conditioning factors for query states prior to softmax normalization. LMA Scaled Mode, which adapts the scaled dot-product formulation while preserving the logit mixing mechanism to mitigate ambiguity under long-context conditions. Through controlled training experiments on an NLP sentiment task, LMA is compared against classical attention. Empirical observations show that the K-Factor variant exhibits training dynamics that are nearly homogeneous with classical attention, despite employing a different internal computation strategy. Additional analysis highlights distinct entropy behavior during training, suggesting alternative information flow characteristics within the model. This paper does not aim to replace classical attention nor claim state-of-the-art performance. Instead, it focuses on mechanism analysis, training behavior, and computational complexity, including discussion of quadratic and linear scaling considerations. The work is intended as: a technical reference for researchers exploring attention variants, a conceptual contribution to logit-level information mixing, and a foundation for further optimization or architectural experimentation. This research is presented as an independent, exploratory study, emphasizing transparency, limitations, and reproducibility over benchmark-driven claims.