A Large-Dimensional Analysis of ESPRIT DoA Estimation: Inconsistency and a Correction via RMT

In this paper, we perform asymptotic analyses of the widely used ESPRIT direction-of-arrival (DoA) estimator for large arrays, where the array sizeNand the number of snapshotsTgrow to infinity at the same pace. In this large-dimensional regime, the sample covariance matrix (SCM) is known to be a poor eigenspectral estimator of the population covariance. We show that the classical ESPRIT algorithm, that relies on the SCM, and as a consequence of the large-dimensional inconsistency of the SCM, producesinconsistentDoA estimates as$N, T →$∞ with$N/T →c$ε (0,∞), for both widely- and closely-spaced DoAs. Leveraging tools from random matrix theory (RMT), we propose an improved G-ESPRIT method and prove itsconsistencyin the same large-dimensional setting. From a technical perspective, we derive a novel bound on the eigenvalue differences between two potentially non-Hermitian matrices, which may be of independent interest. Numerical simulations are provided to corroborate our theoretical findings.