RIS-Assisted Multicast-Unicast Transmission With NOMA: Adaptive User Pairing, Decoding Ordering, and Beamforming

Reconfigurable intelligent surface-assisted non-orthogonal multiple access (RIS-NOMA) transmission is a promising enabler for concurrent multicast and unicast services. Since RIS alters the wireless channel, conventional NOMA user pairing based on channel gain, as well as successive interference cancellation (SIC) decoding ordering, are no longer applicable. To address this challenge, we propose a joint user pairing and decoding ordering scheme for multicast-unicast transmission in RIS-NOMA systems. To this end, we formulate a max-min multicast rate problem under the constraints of unicast quality of service and SIC decoding rates. To tackle the problem with complex non-convex structure, we propose a low-complexity sequential rank-one constraint relaxation-based reformulation-linearization technique successive convex approximation (SROCR-based RLT-SCA) algorithm, which transforms quadratic constraints into first-order formulations. We prove that multi-variable SROCR beamforming design satisfies rank-one constraints. Simulation results demonstrate that the proposed scheme ensures both user fairness and rate fairness, while validating the necessity of jointly optimizing user pairing, decoding ordering and beamforming. Furthermore, multicast rate is up to unicast density and RIS scale: decoding ordering prevails with sparse unicast users or a large RIS, whereas user pairing dominates in densetive unicast-user scenarios with a limited RIS, where the proposed scheme still attains an improvement of up to 35.1% over the scheme of optimal pairing and fixed ordering.