Joint Resource Allocation for Underwater Acoustic Cooperative Communication Networks: A Hierarchical Combinatorial Bandit Approach

This study investigates a joint resource allocation problem for the unknown time-varying underwater acoustic cooperative communication networks. A low-computational-cost and high-efficiency joint strategy decision-making approach, enhanced in three aspects: improving the quantity and quality of learning information, and optimizing the search method for the complex coupled strategy space, is proposed. First, we formulate a combinatorial multi-armed bandit learning model, which offers the separate learning spaces for each sub-strategy. Thus, the player can effectively learn the high-dimensional entire joint strategy by searching the low-dimensional sub-strategy space with low computational cost to avoid the complex direct search in the high-dimensional space. As a result, the solving difficulty caused by the complex coupling of joint sub-strategies can be significantly mitigated. Second, a hierarchical combinatorial bandit learning structure is formulated to improve the quantity and quality of learning information. The estimation layer is constructed to denoise the observed rewards, thereby providing higher quality learning information. The prediction layer is constructed to predict the rewards of all sub-strategies, thereby enriching learning information. Furthermore, in the decision-making layer, we propose an extended upper confidence bound 1 index function to achieve the effective integration of the higher quality denoised learning information and richer predicted learning information to improve the online decision. Finally, the simulation results verify that the proposed innovations can significantly improve the online decision-making efficiency for complex coupled joint strategies and effectively enhance the performance of underwater acoustic cooperative communication networks.