Lighting the Way Forward in Endoscopic Vision: Self-Supervised Approaches to Address Specular Highlights

Computer vision has the potential to enhance endoscopic procedures by enabling capabilities such as lesion detection, instrument navigation, and 3D anatomical modelling. However, irregular light patterns, like specular highlights, pose significant challenges. This thesis addresses these challenges through three key self-supervised solutions, motivated by the lack of ground truth data in endoscopic imaging. First, EndoSTTN, a spatiotemporal GAN, inpaints anatomy hidden by specularities using spatial cues and prior frames, operating unsupervised with automatic highlight detection. It improves vision tasks crucial for 3D reconstruction and camera motion estimation, such as stereo disparity, optical flow, and feature matching. Second, while inpainting handles specularities through preprocessing, this thesis also investigates single-step methods for feature matching and monocular depth estimation. Robustness to specularities is tested and improved using augmentation and pseudo-labels. These tools employ EndoSTTN for inpainting and CycleSTTN for generating temporally consistent, realistic specularities in endoscopic videos. Lastly, monocular depth estimation remains challenging despite these techniques. The third solution, SHADeS, is a self-supervised model leveraging physics-based insights to decouple depth and lighting in colonoscopy scenes. It does this by jointly estimating shading, albedo, depth, and specularities. Unlike methods guided by a Lambertian light model, SHADeS uses a non-Lambertian approach, treating specular reflections separately to robustly estimate depth and light components. All solutions were rigorously evaluated through quantitative and qualitative methods, including comparisons, ablation studies, and tests across diverse datasets, demonstrating their effectiveness and generalizability.