Breaking the Diagonal Mold: Full-Scattering Matrix Control in BD-RIS for Securing Satellite RSMA

Satellite communications (SatCom) face fundamental security challenges due to their inherent broadcast nature. To address this, we exploit beyond-diagonal reconfigurable intelligent surface (BD-RIS) to unleash its full-scattering matrix control for enhanced secure beamforming flexibility in SatCom with rate-splitting multiple access (RSMA), where the satellite attempts to convey private signals to legitimate users with blocked direct downlinks and multiple eavesdroppers. To maximize the worst-case secrecy rate among legitimate users, a max-min fairness (MMF) problem is formulated with imperfect wiretap channel state information (CSI) via joint precoding, RIS configuration, and rate splitting optimization. By using the block coordinate descent (BCD) method, these optimization variables are decoupled with different subproblems and solved by the penalty dual decomposition (PDD) method iteratively. Furthermore, we develop a computationally efficient suboptimal solution that employs diagonal RIS (D-RIS) with reduced hardware and computational complexity, where alternating optimization (AO) and successive convex approximation (SCA) methods are employed to solve the non-convex problem. Simulation results demonstrate that our proposed BD-RIS-RSMA scheme achieves significant performance improvements compared to baseline schemes, while the suboptimal diagonal RIS scheme offers a favorable performance-complexity tradeoff.