Research on Multi-USV Collision Avoidance Based on Priority-Driven and Expert-Guided Deep Reinforcement Learning

Deep reinforcement learning (DRL) has demonstrated considerable potential for autonomous collision avoidance in unmanned surface vessels (USVs). However, its application in complex multi-agent maritime environments is often limited by challenges such as convergence issues and high computational costs. To address these issues, this paper proposes an expert-guided DRL algorithm that integrates a Dual-Priority Experience Replay (DPER) mechanism with a Hybrid Reciprocal Velocity Obstacles (HRVO) expert module. Specifically, the DPER mechanism prioritizes high-value experiences by considering both temporal-difference (TD) error and collision avoidance quality. The TD error prioritization selects experiences with large TD errors, which typically correspond to critical state transitions with significant prediction discrepancies, thus accelerating value function updates and enhancing learning efficiency. At the same time, the collision avoidance quality prioritization reinforces successful evasive actions, preventing them from being overshadowed by a large volume of ordinary experiences. To further improve algorithm performance, this study integrates a COLREGs-compliant HRVO expert module, which guides early-stage policy exploration while ensuring compliance with regulatory constraints. The expert mechanism is incorporated into the Soft Actor-Critic (SAC) algorithm and validated in multi-vessel collision avoidance scenarios using maritime simulations. The experimental results demonstrate that, compared to traditional DRL baselines, the proposed algorithm reduces training time by 60.37% and, in comparison to rule-based algorithms, achieves shorter navigation times and lower rudder frequencies.