Joint Transmit Beamforming and Reflection Optimization for Beyond Diagonal RIS Aided Multi-Cell MIMO Communication

The sixth-generation (6G) wireless networks will rely on ultra-dense multi-cell deployment to meet the high rate and connectivity demands. However, frequency reuse leads to severe inter-cell interference, particularly for cell-edge users, which limits the communication performance. To overcome this challenge, we investigate a beyond diagonal reconfigurable intelligent surface (BD-RIS) aided multi-cell multi-user downlink MIMO communication system, where a BD-RIS is deployed to enhance desired signals and suppress both intra-cell and inter-cell interference. We formulate the joint optimization problem of the transmit beamforming matrices at the BSs and the BD-RIS reflection matrix to maximize the weighted sum rate of all users, subject to the challenging unitary constraint of the BD-RIS reflection matrix and transmit power constraints at the BSs. To tackle this non-convex and difficult problem, we apply the weighted minimum mean squared error (WMMSE) method to transform the problem into an equivalent tractable form, and propose an efficient alternating optimization (AO) based algorithm to iteratively update the transmit beamforming and BD-RIS reflection using Lagrange duality theory and manifold optimization. Numerical results demonstrate the superiority of the proposed design over various benchmark schemes, and provide useful practical insights on the BD-RIS deployment strategy for multi-cell systems.