Adaptive Machine Learning Framework for Real-time Optical Coherence Tomography Artefact Correction in Retinal Detachment Surgery: A Hybrid Convolutional Neural Network–Long Short-term Memory–Transformer Approach with Unsupervised Domain Adaptation

Abstract Background: Intraoperative optical coherence tomography (OCT) provides high-resolution cross-sectional imaging during retinal detachment surgery, but motion artefacts, speckle noise and dynamic retinal changes often limit its use. These artefacts degrade image quality and complicate surgical decision-making, emphasizing the need for reliable real-time correction to improve outcomes. Methods: We developed a hybrid adaptive machine-learning framework for real-time OCT artefact correction. The system integrates convolutional neural networks (CNNs) for spatial analysis, long short-term memory (LSTM) networks for temporal sequence consistency and a lightweight transformer module for efficient feature prioritization. To overcome the lack of labelled intraoperative datasets, unsupervised domain adaptation transfers knowledge from pre-operative to intraoperative conditions. The framework was implemented with FPGA acceleration, achieving sub-200 ms latency suitable for surgical workflows. Results: The proposed method outperformed baseline approaches, achieving a peak signal-to-noise ratio of 32.7 dB and a structural similarity index of 0.921. It preserved retinal layer continuity and achieved higher surgical relevance scores compared with CNN-LSTM and transformer-based methods. In a prospective clinical study of 20 retinal detachment surgeries, its use reduced unplanned manoeuvres by 23%, shortened procedure times by 18% and achieved higher reattachment rates (94% vs. 82%, P < 0.05). Surgeons also reported greater confidence in identifying residual traction and vitreous remnants. Conclusion: This hybrid adaptive framework provides robust real-time OCT artefact correction, enhances intraoperative visualization and improves anatomical and functional outcomes in retinal detachment surgery. Its modular and adaptive design supports broader applications in ophthalmic surgery and other intraoperative imaging modalities.