Precoding and Rate-Splitting Design for Beyond-Diagonal RIS-Aided ISAC Systems With RSMA

This paper studies a beyond-diagonal reconfigurable intelligent surface (BD-RIS)-aided integrated sensing and communications (ISAC) system enhanced by rate-splitting multiple access (RSMA) to improve sensing and communication abilities with flexible interference management. Unlike conventional RIS, BD-RIS is a generalized RIS paradigm with a non-diagonal scattering matrix, offering additional degrees of freedom. Moreover, RSMA is a novel multiple access framework for effective and flexible interference control. In this work, we formulate a sensing signal-to-noise ratio maximization problem with reduced multi-user interference (MUI), which is challenging to jointly design the BD-RIS scattering, precoding, and rate splitting due to strong cross-domain coupling. To address these issue, we devise a two-stage optimization framework. First, the BD-RIS scattering matrix is optimized via symmetric unitary projection to improve the channel gain, where a closed-form solution is derived. Then, the weighted minimum mean square error framework is customized for precoding and rate splitting optimization. Simulation results validate the advantages of the proposed system and the algorithm, showing that the BD-RIS-aided RSMA-enhanced ISAC system is able to effectively improve the sensing performance and flexibly control the MUI.