Can Vision Language Models Zero-Shot Thermal Images for Private and Robust Human Robot Interaction?

Vision language models (VLMs) have shown strong performance in RGB-based human-robot interaction (HRI). However, using RGB cameras in homes faces challenges due to privacy issues and poor performance in the dark, such as during night-time elderly care. Thermal imaging offers a privacy-preserving alternative that works without light. This raises a natural yet previously unexplored question: Can general-purpose VLMs effectively interpret thermal images in a zero-shot manner for privacy-preserving and robust HRI? In this work, we conduct a thorough evaluation of this capability using a real-world dataset. Specifically, we investigate whether the latest VLMs are reliable for safety-critical HRI tasks. We benchmark six leading VLMs on the OctoNet dataset, which contains 975 thermal sequences. To avoid self-evaluation bias, we use an ensemble of three independent large language models to score the predictions and measure stability. Our results reveal a critical performance disparity: while VLMs are accurate on large body movements (e.g., Sitting: 92.8%), they struggle on fine-grained interactions (e.g., Hand Gestures: