Fractional Calculus for Large Language Models: A Survey of Potentials, Mechanisms, and Challenges

Large language models (LLMs) face critical bottlenecks like the “memory wall” and context degradation due to standard integer-order Markovian assumptions. This survey reviews integrating fractional calculus to explore potential solutions. Synthesizing the recent literature reveals a transitional landscape: while fractional mechanisms yield significant empirical gains in medium-scale architectures (e.g., up to 14.36% absolute accuracy improvements), explicit validation on autoregressive models exceeding one billion parameters remains largely unexplored. By introducing heavy-tailed power-law memory and non-local historical gradients, fractional operators help mitigate catastrophic forgetting and optimization instability. Furthermore, we propose dual-integration frameworks: a decoupled neuro-symbolic architecture for scientific computing, and an endogenous fractional evolution architecture for native long-context reasoning. While fractional dynamics offer mathematical pathways to alleviate LLM constraints, computational overhead and training stability challenges remain. Future research should prioritize selective fractionalization and data-driven scientific discovery.