AdaptFly: Prompt-Guided Adaptation of Foundation Models for Low-Altitude UAV Networks

Low-altitude Unmanned Aerial Vehicle (UAV) networks rely on semantic segmentation as a core perceptual capability enabling distributed sensing–communication–control co-design across heterogeneous agents. However, segmentation foundation models deteriorate quickly under weather, lighting, and viewpoint drift. Resource-limited UAV cannot run gradient-based test-time adaptation, while resource-massive UAV adapt independently, wasting shared experience. To address these challenges, we propose AdaptFly, a prompt-guided test-time adaptation framework that adjusts segmentation models without weight updates. AdaptFly features two complementary adaptation modes. For resource-limited UAV, it employs lightweight token-prompt retrieval from a shared global memory. For resource-massive UAV, it uses gradient-free sparse visual prompt optimization via Covariance Matrix Adaptation Evolution Strategy, with offline prompt distillation for knowledge sharing. An activation-statistic detector triggers adaptation, while cross-UAV knowledge pool consolidates prompt knowledge and enables fleet-wide collaboration with negligible bandwidth overhead. Extensive experiments on UAVid and VDD benchmarks, along with real-world UAV deployments under diverse weather conditions, demonstrate that AdaptFly significantly improves segmentation accuracy and robustness over static models and state-of-the-art TTA baselines. The results highlight a practical path to resilient, communication-efficient perception in the emerging low-altitude economy.