Enabling Software-Defined Tiered LLM Inference Continuum on Passive Optical Network

The rapid expansion of large language models (LLMs) has shifted inference workloads from centralized cloud servers toward distributed execution across fog and edge environments. As this trend accelerates, existing broadband infrastructures, particularly passive optical networks (PONs), must evolve to meet the stringent latency, bandwidth, and coordination demands of real-time LLM inference. Therefore, it is essential to enhance the access network with intelligence and programmability to support efficient, scalable, and low-latency inference delivery across multiple network layers. This paper presents a software-defined architecture for enabling tiered LLM inference over Time and Wavelength Division Multiplexed Passive Optical Networks (TWDM-PON). The proposed system incorporates LLM-integrated Optical Line Terminals (OLTs) and Optical Network Units (ONUs), each equipped with processing and queuing capabilities to execute portions of LLM inference tasks locally. These intelligent nodes collaborate under centralized Software-Defined Networking (SDN) control to manage inference-related traffic and optimize resource allocation across the network. Through inference-aware dynamic bandwidth allocation, time and wavelength slicing, and queue-level traffic isolation, the architecture prioritizes LLM traffic to provide low and consistent latency for LLM inference tasks for prompt and token streams without degrading conventional broadband services. Simulation results show that the proposed architecture reduces average inference latency by up to 50%, improves throughput by up to 7%, and lowers jitter and packet drop probability by up to 32% and 25.01%, respectively, while maintaining end-to-end Quality of Service (QoS) under mixed broadband and LLM traffic.