Linear Attention-Driven DRL With Sparse Expert Fusion: A Dynamic Optimization Algorithm of Underwater Manipulator for Long Horizon Tasks

Deep Reinforcement Learning (DRL), leveraging the nonlinear approximation capabilities of deep neural networks, maps high-dimensional perception information to robotic control commands and has been widely applied to various continuous control tasks. However, for long-horizon tasks, manipulators often face challenges such as high-dimensional exploration spaces and sparse rewards, making it difficult to learn effective strategies and even leading to potentially dangerous actions. Additionally, collecting a large volume of high-quality expert demonstrations for underwater manipulators is challenging. To overcome these limitations, this study proposes a DRL algorithm that integrates a small amount of expert experience for long-horizon control of underwater manipulators. Firstly, a DRL dynamic optimization strategy based on expert experience is designed, featuring a dual-buffer dynamic sampling mechanism that enables efficient early-stage learning. Secondly, a linear attention mechanism network is developed to aggregate global task features while maintaining low computational complexity, allowing the model to effectively process high-dimensional sensory inputs and share features across multiple tasks. Furthermore, a staged reward function is designed to steadily learn skills for each phase, ultimately completing the entire long-horizon task. To validate the effectiveness of the proposed algorithm, an underwater simulation environment is constructed in Gazebo. In this simulation environment, the manipulator is trained on various long-horizon tasks, including grasp-place, grasp-stack, and grasp-insert. Training results demonstrate that the proposed algorithm outperforms existing methods in terms of task success rate and policy stability. Additionally, real-world experiments in a water tank environment verify the algorithm’s generalization and robustness in practical underwater operations.