Integrated Sensing and Communication for Underwater Acoustic Networks Based on Deep Reinforcement Learning

This paper investigates a new integrated sensing and communication (ISAC) scheme for underwater acoustic (UWA) networks based on deep reinforcement learning, referred to as Deep UWA-ISAC (DeepUSC). Specifically, we consider a UWA-ISAC system, where an autonomous underwater vehicle (AUV) transmits the collected environmental data to the buoy, while sensing the sea area to monitor the unauthorized mobile target. The expected communication rate over a given navigation period is maximized by jointly optimizing the AUV's beamforming and trajectory, subject to the constraints on the average signal-to-noise ratio requirement for target sensing as well as the navigation mission, collision avoidance, and maximum transmit power limit of the AUV. Three key challenges for DeepUSC are: (i) long propagation delays in the UWA-ISAC system may cause interference from the previous echo to the current ISAC signal; (ii) the mobility pattern of the target is unknown in advance; and (iii) the AUV navigation-oriented ISAC problem is a long-term optimization problem as the navigation mission typically lasts for a long period. To circumvent the above challenges, DeepUSC is developed based on a specific partially observable Markov decision process model termed episode task, where each navigation period is considered as an episode and the navigation mission corresponds to the episode task. Through judicious design of a reward function and action selection policy, DeepUSC can satisfy various preset constraints without requiring prior knowledge of the target's mobility. Besides, to enable efficient learning in episode tasks, we propose an episodic experience replay mechanism that dynamically prioritizes high-value recent experiences and utilizes all experiences generated within each episode to jointly train the neural network. Simulation results demonstrate that compared with benchmarks, DeepUSC yields a higher communication rate while satisfying all constraints, converges faster, and is more robust against different simulation setups.