Joint Active and Passive Beamforming for Multi-UE Communication and Extended Target Detection in IRS-Assisted ISAC Systems

Intelligent reflecting surface (IRS)-assisted integrated sensing and communications (ISAC) systems have been extensively studied to meet higher sensing requirements. For detection-oriented IRS-assisted ISAC problems, most studies have overlooked the detection interference caused by clutters and modeled simplified point-like targets. This paper investigates extended target detection in IRS-assisted ISAC systems within clutters. We present an optimal generalized likelihood ratio test detector and derive the corresponding probability of detection (PD) and probability of false alarm in closed form. Then, we jointly optimize the active and passive beamforming of the base station and IRS to maximize the PD under multi-user equipment (UE) communication rate constraints and the total transmit power constraint. We first simplify the complex objective function by proving the invariant property of a subspace projection matrix. We then present a novel alternating optimization (AO)-based algorithm to decouple the original problem into two subproblems, consequently convexified and solved using the semidefinite relaxation method. Simulations demonstrate the convergence of the proposed algorithm. The PD performance and the communication and sensing trade-off are significantly improved, compared to benchmarks.