Optimal Resource Allocation in Relay-Assisted ISAC Systems via Deep Reinforcement Learning

This paper studies joint beamforming and power allocation in a full-duplex decode-and-forward (FD-DF) relay-assisted monostatic integrated sensing and communication (ISAC) system. The base station performs monostatic sensing in the presence of clutter, residual self-interference (SI), and uplink transmissions, while it must satisfy downlink and uplink quality-of-service (QoS) requirements under total power constraints. Unlike prior FD-ISAC or relay-aided works that neglect monostatic sensing geometry, explicit clutter modeling, or dynamic environments, we formulate a nonconvex optimization problem capturing coupled beamforming vectors, user powers, and sensing signal covariance, with end-to-end DF-relay rate bottlenecks and SI-aware sensing signal-to-interference-plus-noise ratio (SINR) maximization. To support real-time adaptation in dynamic environments, we model the design as a Markov decision process (MDP) and apply a deep deterministic policy gradient (DDPG) algorithm with a tailored state representation (dynamic user/echo channels), continuous actions (beamformers and powers), and a constraint-aware reward that combines piecewise-linear sensing SINR gains above 5 dB with normalized QoS violation penalties. We study several system setups, including non-relay, half-duplex (HD) relay, and FD relay-assisted configurations, and show that the proposed FD relay-assisted ISAC scheme achieves better overall performance. Extensive simulations indicate that the proposed DDPG-based policy achieves comparable or improved average performance relative to a successive convex approximation (SCA) benchmark under the considered system settings, including up to 0.7 dB higher mean sensing SINR, substantially reduced performance variance under strong self-interference, and near-100% QoS satisfaction across diverse channel realizations, while enabling lower online inference complexity. These results highlight the practical effectiveness of deep reinforcement learning (DRL) for resource allocation in relay-enhanced ISAC networks.