Seeing with Partial Certainty: Conformal Prediction for Robotic Scene Recognition in Built Environments

In assistive robotics for people with disabilities (PWD), accurate place recognition is vital for safe and effective indoor navigation. Language interfaces powered by large language models (LLMs) and vision-language models (VLMs) show promise but are prone to hallucinations and ambiguity in interpreting human instructions. We propose seeing with partial certainty (SwPC), a lightweight uncertainty alignment framework that enhances VLM-based place recognition. SwPC uses conformal prediction to detect when the model is uncertain and selectively request human assistance, reducing unnecessary interventions. It requires no model fine-tuning and can be applied directly to existing VLMs. Experiments on the Matterport3D dataset demonstrate that SwPC improves success rates and reduces human involvement compared to prior methods. SwPC offers a scalable solution for integrating uncertainty modeling into foundation models, supporting more robust and trustworthy human–robot interaction in complex built environments.