RT-ZSDR: A real-time zero-shot segmentation and dense reconstruction framework for plant phenotyping robot

Annotation scarcity, poor model generalization and lagged data processing remain key bottlenecks hindering the practical deployment of phenotyping robots. To address these issues, we developed a novel phenotyping robot capable of online 3D reconstruction and zero-shot segmentation directly on the edge. Diverging from conventional semantic SLAM, our core contribution is RT-ZSDR, a framework featuring two key methodological novelties. First, we introduce the ForeCut pipeline for target extraction, which innovatively fuses DINO features with 3D geometric spatial information, leveraging multi-view semantic-spatial consistency to achieve annotation-free, zero-shot dense segmentation and reconstruction. Second, we designed a hardware-coupled loop closure strategy utilizing the robotic arm's kinematic feedback as prior constraints to significantly improve loop closure recall. Supported by edge computing Jetson Orin NX, the tracking and segmentation process takes approximately 0.24 s per frame after an initialization period of 1.82 s. RT-ZSDR's phenotypic measurements demonstrated strong correlations with reference baseline in both laboratory settings (n=90, PlantEye measurements as reference baseline; R 2 =0.990, 0.939, 0.725, and 0.861 for plant height, projected leaf area, surface area, and volume) and practical greenhouse environments (n=48, manual measurements as reference baseline; R 2 =0.965, 0.862 for plant height and stem diameter). Additionally, evaluated against COLMAP benchmarks (n=24), the system achieved a mean 3D reconstruction F1-score of 0.816.