Hybrid Dual-Path Linear Transformations for Efficient Transformer Architectures

Standard Transformer architectures rely heavily on dense linear transformations, treatingfeature projection as a monolithic, full-rank operation. We argue that this formulation isinefficient and lacks the structural inductive bias necessary for distinguishing between localfeature preservation and global context integration. To address this, we introduce the HybridDual-Path Linear (HDPL) operator, which decomposes the affine transformation intotwo topologically distinct pathways: a sparse block-diagonal component for high-rank localprocessing, and a low-rank Variational Autoencoder (VAE) bottleneck for global contextregularization.By “surgically” replacing specific projections (Query, Key, Value, Gate, Up) with HDPLoperators while retaining standard dense layers for aggregation (Output, Down), we achieve asuperior balance of efficiency and representational power. Experiments on the FineWeb-Edudataset demonstrate that the HDPL architecture outperforms a standard Llama-style baseline,reducing validation loss while simultaneously reducing parameter count by ≈ 6.8%. Beyondimmediate performance gains, we discuss how the explicit materialization of a probabilisticlatent space within the Transformer backbone serves as a vital architectural affordance, offer-ing new pathways for inference-time or hypernetwork induced control, continual adaptation,interpretability, and cross-model or cross-modal synchronization. The code is available at https://github.com/VladimerKhasia/HDPL