Where We Hit the Limits: Blue-Sky Directions for Large Language and Multimodal Models

Despite the revolutionary impact of the Transformer architecture and the "Attention Is All You Need" paradigm, the field is currently facing a performance plateau characterized by diminishing returns on scaling and the computational bottlenecks of quadratic attention complexity . This talk explores "blue-sky" directions to overcome these limitations through innovations in model architectures and techniques. First, we address the instability issues inherent in current linear attention mechanisms, such as DeltaNet, which suffer from error accumulation due to Euler discretization . We introduce Error-Free Linear Attention (EFLA), a novel framework that derives an exact solution to the underlying continuous-time ordinary differential equation (ODE) . By leveraging Rank-1 matrix dynamics, EFLA achieves a computationally tractable, linear-time update rule that eliminates discretization error, offering guaranteed stability and superior performance in language modeling tasks. Second, we examine the intersection of vision-language reasoning and robotic control (VLA models), identifying the limitations of Supervised Fine-Tuning (SFT) and the scalability challenges of reinforcement learning in robotics . We present NORA- 1.5, a VLA architecture that couples an auto-regressive backbone with a Flow- Matching Action Expert . By utilizing World Model-guided rewards and Direct Preference Optimization (DPO), NORA-1.5 enables scalable post-training on synthetic data, achieving state-of-the-art results on benchmarks like SimplerEnv and LIBERO while demonstrating robust real-world deployment on the Galaxea A1 robot . These contributions collectively chart a path toward more efficient, stable, and capable AI systems."