CAWN: Continuous Acoustic Wave Networks for Autoregressive Language Modeling

Read full text Summarise with VIRAT
Authors: Dejan Čugalj, Aleksandar Jevremovic
Cited by: 0
References: 21

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

Modern Large Language Models (LLMs) rely heavily on the Transformer architecture and the self-attention mechanism, which scales quadratically with sequence length. Recent alternatives, such as State Space Models (SSMs), which seek to achieve linear time complexity, often suffer from signal degradation over extended contexts. In this paper, we introduce a novel, fully continuous sequence-mixing architecture: the Continuous Acoustic Wave Network (CAWN). Instead of discrete matrix-based attention, our architecture projects hidden states into multi-headed complex-domain phasors. Sequence mixing is achieved through a causal, O(L) Phase Accumulation mechanism. To prevent signal degradation over ultra-long contexts, we introduce a dual-gated Selective Phase Resonance mechanism. This incorporates Frequency-Dependent Retention, Hard-Threshold Gating via Straight-Through Estimation, and a Temporal Syntax Cache (1D-Convolution) to capture short-term local dependencies before global wave projection. Furthermore, we replace standard dense linear projections with a Depth-wise Harmonic Convolution for optimal spatial frequency mixing, augmented by Block Attention Residuals to enable depth-wise state routing. We scale this architecture to a 150-Million-parameter model, utilizing custom Triton kernels for hardware-efficient, true-complex phase accumulation in float32. Trained via a continuous streaming loop on a 100-Billion-token English corpus, the prototype is evaluated at a 5-Billion-token milestone. Subjected to a Targeted Semantic Retrieval protocol, our empirical evaluations demonstrate robust vocabulary acquisition and highly extended explicitly learned contextual denoising. By leveraging O(1) state-passing via chunked prefill, the model successfully retrieves targeted information across 2,000,000 tokens while strictly plateauing at 8.72 GB of Peak VRAM, empirically overcoming the O(L^2) context memory wall.

Fields of Study & Subjects

Topic Modeling Natural Language Processing Techniques Speech Recognition and Synthesis

Publication details

Journal / source
arXiv (Cornell University)
Publication date
30 March 2026
Language
EN
Document type
Preprint