Multi-Modal Interactive Control of Robotic Arm Based on Offline Large Language Models (Student Abstract)

Large Language Models (LLMs) have revolutionized the modern society significantly with the numerous advanced interactions between humans and AI agents, whereas the usage of most large language models including ChatGPT are not friendly open-sourced and must require users paying a lot for such AI service continuously. Therefore, deploying open-sourced large language models on local servers can be considered as an efficient method to design and implement creative embodied AI algorithms with lower cost and more stable free usage. Inspired by this ordinary motivation, we originally propose and implement the “Socratic Models-ChatGLM”, which is a well-performed algorithm for multi-modal interactive control of robotic arm based on offline large language models via the facile PyBullet platform, even presents extraordinary potential to address complicated text-image multi-step long-horizon robotic manipulation tasks.