CAWN: Continuous Acoustic Wave Networks for Autoregressive Language Modeling
Summary
No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.
Summarise with VIRAT Free to try with your Science Catalogs account.Abstract
Modern Large Language Models (LLMs) rely heavily on the Transformer architecture and the self-attention mechanism, which scales quadratically with sequence length. Recent alternatives, such as State Space Models (SSMs), which seek to achieve linear time complexity, often suffer from signal degradation over extended contexts. In this paper, we introduce a novel, fully continuous sequence-mixing architecture: the Continuous Acoustic Wave Network (CAWN). Instead of discrete matrix-based attention, our architecture projects hidden states into multi-headed complex-domain phasors. Sequence mixing is achieved through a causal, O(L) Phase Accumulation mechanism. To prevent signal degradation over ultra-long contexts, we introduce a dual-gated Selective Phase Resonance mechanism. This incorporates Frequency-Dependent Retention, Hard-Threshold Gating via Straight-Through Estimation, and a Temporal Syntax Cache (1D-Convolution) to capture short-term local dependencies before global wave projection. Furthermore, we replace standard dense linear projections with a Depth-wise Harmonic Convolution for optimal spatial frequency mixing, augmented by Block Attention Residuals to enable depth-wise state routing. We scale this architecture to a 150-Million-parameter model, utilizing custom Triton kernels for hardware-efficient, true-complex phase accumulation in float32. Trained via a continuous streaming loop on a 100-Billion-token English corpus, the prototype is evaluated at a 5-Billion-token milestone. Subjected to a Targeted Semantic Retrieval protocol, our empirical evaluations demonstrate robust vocabulary acquisition and highly extended explicitly learned contextual denoising. By leveraging O(1) state-passing via chunked prefill, the model successfully retrieves targeted information across 2,000,000 tokens while strictly plateauing at 8.72 GB of Peak VRAM, empirically overcoming the O(L^2) context memory wall.
Fields of Study & Subjects
Topic Modeling
Natural Language Processing Techniques
Speech Recognition and Synthesis
Publication details
- Journal / source
- arXiv (Cornell University)
- Publication date
- 30 March 2026
- Language
- EN
- Document type
- Preprint