Automatic control of multimodal fusion robot using YoloV8 with CNN

A multimodal data fusion platform that can manage heterogeneous data may be challenging to manually construct as sensory-feedback increases in volume and complexity. Recent years have seen a rise in the study of multi-modal machine learning, with researchers concentrating on methods for processing visual and auditory data. From a robotics standpoint, nevertheless, haptic feedback from environmental interactions is crucial for carrying out practical activities. The capacity to use deep reinforcement learning without knowing the system inside and out is one of its main benefits. One of the most difficult aspects of mobile robotics is path planning, namely avoiding obstacles. An array of human-machine sensors, including four force sensors, two joysticks, and a depth-sensing camera, was used to detect human intent. Instead of using the hands to operate the walker, users may just move or turn around due to the motion controller's ability to understand their purpose via the interactive force. To ensure the algorithm works, we do preliminary tests on healthy people and virtual patients. The results demonstrate that, in comparison to the conventional proportional controller, the suggested motion control method is more precise, user-friendly, and capable of recognizing the user's purpose.To enhance the accuracy of intention identification, a multimodal integration method was suggested, combining the YoloV8 model with Convolutional Neural Networks (CNN). Next, the system used the video camera, force gauges, and joysticks to obtain the moving angle (E), pelvic pose (F), and motion vector (H), accordingly. It is used extraction of features along with data combination to classify the goals, and finally, it outputted the speed control for the motor through the robot's kinematics, using a visual Simultaneous Localization and Mapping (SLAM) model and Deep Reinforcement Learning (DRL). That the system can provide walking aid and that the suggested way is efficient were both shown by the findings.