Techniques for physical layer equalization and monitoring in radio-over-fiber: DSP and machine learning perspectives

• DSP and ML techniques are analyzed for RoF physical layer equalization and monitoring • Bibliometric mapping reveals key research trends and conceptual clusters in RoF • ML is increasingly applied to mitigate high-frequency RoF physical impairments • Lack of standardized datasets hinders benchmarking and fair validation of methods • Future research opportunities are discussed toward intelligent RoF for beyond-5G/6G Radio-over-fiber (RoF) systems are essential to the convergence of optical and wireless networks, serving as key enablers for future high-capacity broadband and 5G/6G applications. However, scaling capacity requires effective strategies for channel equalization and impairment mitigation to overcome complex nonlinearities, particularly at high frequencies. Traditional digital signal processing (DSP) methods are nearing their limits in dynamic scenarios. Thus, this paper addresses a significant lack in the literature by providing a comprehensive bibliometric and systematic analysis to objectively define the evolving roles of established DSP and emerging machine learning (ML) within this critical domain. By analyzing the co-evolution of key research themes, our approach confirms that the field is characterized by a structural coexistence between the two paradigms, rather than simple replacement. The primary results demonstrate that ML techniques, particularly deep learning, establish functional superiority in compensating complex, high-order impairments in high-frequency bands. However, we identify a critical research gap in intelligent monitoring; while ML has high potential for multi-parameter estimation, it remains a niche interest often handled separately across optical and wireless layers. This drives the most pragmatic solution: hybrid DSP+ML architectures. These architectures balance DSP’s computational efficiency, which sets a demanding benchmark for resource consumption, with ML’s intelligence to achieve superior performance under strict cost, size, and power (C-S-P) constraints. The main finding is that this integration is transforming the RoF physical layer, shifting the focus from purely corrective equalization toward pblackictive, system-level control. The next critical challenges for commercial deployment are the successful implementation of real-time hardware and the standardization of public datasets.