Sense, think, and robot control: Multimodal material recognition sensing integrated with large language model reasoning

Touch is fundamental to human material perception, involving diverse receptors and complex cognitive processing. However, existing studies are often limited to single-modal sensing and stop at recognizing material type. In this work, multimodal sensing is integrated with a large language model (LLM) to improve material recognition accuracy and enable deeper thinking. The proposed compact flexible multimodal sensor can simultaneously acquire pressure, thermal, and dielectric signals, achieving an overall recognition accuracy of up to 99.13% across 23 representative materials. Based on the recognition results and task requirements, the LLM generates robot control decisions. Augmented reality (AR) provides system visualization and user control, enabling intuitive human–robot interaction. This system demonstrates intelligent robotic sorting driven by material recognition in real-world scenarios and may inspire the application of multimodal sensors in embodied intelligence.