Deep Learning-Based Assistive Robotic Grasping System for Visually Impaired Individuals

Robotic assistive systems play an important role in enhancing quality of life of visually impaired individuals with greater independence. A significant amount of research has been conducted in assistive grasping systems in recent years, but certain challenges still exist including background noise during voice-based interaction, object occlusion, reliable and safe object handover human robot interaction (HRI), and multimodal interfaces. To address these challenges, this research presents an intelligent assistive robotic grasping framework that integrates the vision module, voice module and robot control module. The vision module includes the enhanced YOLOv11s-seg model that performs unified object segmentation with pose estimation and achieved mAP@50-95 of 0.933 for detection and 0.882 for segmentation with the least computational cost (32.8 GFLOPs) and fast inference time (4.2 ms/image) compared to YOLOv8s and YOLOv9c. The voice module includes a hybrid band-pass with spectral filter that efficiently removes the background noise (low, medium and higher background noise) to convert voice command into texts with accuracy of 93.5%. Additionally, an SVM-based machine learning text classification system is employed to enhance safety by rejecting voice commands unrelated to grasping or object picking, achieving 97.93% accuracy. Finally, the robot control module is introduced with object grasping and safe handover algorithms. The proposed intelligent assistive robotic grasping framework is validated through different experiments which demonstrate the overall grasp success rate of 94.7% across thirteen daily-life objects with minimal execution time of 42.67 s from the voice command to the system to handing over the grasp object to user by the robotic arm. The system ensures operational safety through workspace boundary constraints and built-in force sensing. The proposed approach provides a novel, efficient, and human-centric solution for voice-guided assistive robotic grasping for visually impaired users.