Self-hosted multimodal large language models for speech-driven perception and navigation in construction robotics

This paper explores self-hosted, open-weight multimodal large language models (LLMs) as a speech–vision interface for robot perception and navigation. Using structured function calling, the approach enables transparent control without application-specific training. Two data-local deployment modes are compared: a smaller on-robot model and a larger edge-server model. Both convert spoken commands and images into structured function calls for image capture, analysis, and waypoint navigation. Laboratory evaluations show that the edge-server model outperforms the on-robot model in speed and reliability, particularly in visual tasks, while the on-robot model performs competitively in simple workflows. Findings support hybrid deployment strategies that combine edge-hosted and on-robot models to balance performance and resilience, enabling responsive, cloud-independent robotic operation in construction environments. • Self-hosted framework eliminates cloud dependency while preserving data locality. • Task complexity determines suitable deployment mode for reliable robotic control. • Structured function calls enable transparent, verifiable robot control. • Edge-server model outpaces on-robot model despite local network communication overhead. • Zero-shot deployment requires no task-specific training or demonstration data.