Attention-Based Hierarchical-DRL With Mask for Multi-Timescale Caching, Association, and Secure Content Delivery in UAV-Enabled ISAC Networks

In this research, we investigate the long-term secure content delivery problem in unmanned aerial vehicle (UAV)-enabled integrated sensing and communication (ISAC) networks. We consider that ISAC-assisted UAVs are allowed to store user-requested contents, provide content delivering service to users, and perform eavesdropper detection. However, the openness of UAV-enabled networks makes the content delivery network more susceptible to security threats. To address the eavesdropper detection, we propose a Cramér-Rao Lower Bound (CRLB) and an extended Kalman Filter (EKF)-based location estimation algorithm. We then examine the secrecy throughput of users and formulate the joint user association, UAV deployment, content caching, communication, and sensing beamforming problem as a long-term secure throughput maximization problem. As the formulated problem is a mixed-integer non-linear programming problem (MINLP) and cannot be solved conveniently, we decompose it into two subproblems, namely, a long-timescale content caching subproblem, and a short-timescale user association, UAV deployment, communication and sensing beamforming subproblem. To solve the subproblems, we transform it into a Markov decision process (MDP) and we propose an attention-based hierarchical deep reinforcement learning (HDRL) with an action mask and design a double deep Q-network (DDQN) algorithm to obtain the long-timescale and an attention-based DDQN with an action mask for short-timescale strategies. Specifically, we first obtain a long-timescale strategy for content caching. Given the long-timescale strategy, we then obtain the short-timescale user association, UAV deployment, communication, and sensing beamforming strategy. Simulation results demonstrate the effectiveness of the proposed algorithms.