AlperYildirim1/Pay-Attention-Later: Parameter Efficient PRISM

Release v3.2.0 Added parameter efficient architecture to make weight tying possible. Release v3.1.0 📖 Documentation Text This section contains the "Physics Validation Suite" for the RoSE architecture. Instead of just measuring translation accuracy (BLEU), these tests diagnose how the model thinks. We treat tokens not as static numbers, but as complex-valued waves that can interfere and resonate. We use two specific audits to prove the architecture is working: The "Resonance" Test (Phase Locking) The Question: How does the model connect related concepts? The Test: We measure the physical phase distance between a target word (e.g., "Leiter") and its semantic trigger (e.g., "Firma"). The Result: We observe a "Funnel Effect." In early layers, the relationship is loose. As the network deepens, the target word physically locks onto its trigger (Resonance), reducing the phase error significantly. This proves the model uses phase synchronization to bind concepts. The "Gating" Audit (Amplitude Dynamics) The Question: Is the model using the Gating mechanism correctly? The Test: We inspect the magnitude (|z|) of the vectors across all layers to see if they collapse to a fixed value. The Result: We confirm that the model does not force everything to the unit circle (1.0). Instead, it uses Amplitude as a "Volume Control"—amplifying confident predictions (High Energy) and silencing unrelated noise (Low Energy) via the ModReLU gates. Release v3.0.0: Gated Spectral Layers and Rotary Semantic Embeddings (RoSE) This version introduces significant advancements in the PRISM architecture, transitioning from static spectral mixing to dynamic, gated spectral convolutions. This update establishes a new framework for token representation through Rotary Semantic Embeddings (RoSE), providing a mathematically grounded alternative to standard attention mechanisms (O(N log N) vs O(N^2)). 🔹 Core Scientific Contributions Rotary Semantic Embeddings (RoSE) We introduce RoSE, our new embedding paradigm where tokens are represented as complex-valued waveforms with an Intrinsic Phase. Semantic Phase Separation: Empirical analysis (e.g., Polysemy Density Tests) demonstrates that the model learns to encode semantic distinctions (polysemy) via phase rotation, creating a distinct "high-rotation regime" for ambiguous terms compared to monosemous controls. Interferometric Identity: This phase-locking mechanism allows tokens to maintain their structural identity during global FFT-based mixing, preventing the spectral aliasing commonly found in real-valued architectures like FNet. Cartesian Gated Spectral Convolutions The architecture now features a data-dependent gating mechanism, allowing for input-aware frequency modulation. Dynamic Filtering: By utilizing Cartesian gates (sigmoid-activated projections on both real and imaginary axes), the model filters frequency-domain information dynamically, exhibiting asymmetric steering of real and imaginary components. Computational Efficiency: PRISM maintains O(N log N) complexity, offering a scalable solution for long-sequence modeling (tested up to 4096 tokens) without the quadratic memory overhead of Multi-Head Attention. Harmonic Stability The integration of Phase-Preserving Activation (ModReLU) and Phase-Preserving Layer Normalization (PPLN) ensures that the structural information encoded in the phase is preserved across deep layers, while the semantic energy (magnitude) is modulated to represent confidence. 📈 Benchmarks & Validation Evaluated on the WMT14 DE-EN translation task: SacreBLEU Score: ~27.569029 COMET Score: 0.796917 🛠 Technical Specifications Encoder: Hybrid Architecture (5-Layer Gated Spectral PRISM + 1-Layer Attentional Refinement) Decoder: 6-Layer RoPE-equipped Autoregressive Transformer Complexity: Log-Linear O(N log N) Embedding Dimension: 512 (Complex-valued) 📝 Usage & Integration The model is self-contained in modeling_prism_gated.py and supports the standard Hugging Face initialization protocol. Yujivus/PRISM-Molecule-100k from modeling_prism_gated import PRISMHybrid_RoPE model = PRISMHybrid_RoPE( vocab_size=58101, d_model=512, num_encoder_layers=6, # Spectral Layers num_refining_layers=0, # Attentional Layer num_decoder_layers=6, architecture="PRISM_Molecule" )