UniSurg: A Unified Multitask Framework for Robotic Surgical Scene Understanding

Surgical scene understanding is a vital intelligent technique in robot-assisted surgery, including surgical instrument detection, segmentation, and instrument–tissue interaction detection. Existing methods typically address these tasks in isolation, neglecting the intrinsic correlations among them. In this work, we innovatively propose a unified multitask framework named UniSurg, being the first to jointly address these three critical aspects of surgical scene understanding, thereby providing the robot with multidimensional perceptual capabilities. By exploring the inter-task correlations and reusing shared features, UniSurg has been demonstrated to significantly enhance the scene analysis performance. To address pose variability of the instruments under the constrained field of view in laparoscopic surgery, we design an Attention Enhanced Conditional Convolution (AEC-Conv) that dynamically adjusts kernels based on pose-specific features for improved adaptability. To further enhance interaction detection, we propose the Temporal Difference Enhancement module (TDE), which captures motion cues by amplifying inter-frame differences, and the Pyramid Global Feature Enhancement module (PGFE), which leverages graph-based hierarchical context to model global relational dependencies. Experiments on the Endovis2018 dataset and a clinical multitask dataset MILVis demonstrate the superior multitask performance of UniSurg.