Exploring ultralight dark matter self-coupling via the gravitational wave background

Abstract Supermassive black hole (SMBH) binary mergers are a primary source of the stochastic gravitational wave background (SGWB) in the nanohertz band, now being actively probed by pulsar timing arrays (PTAs). We investigate how ultralight dark matter (ULDM) with a quartic self-interaction, forming solitonic cores around SMBHs, modifies the SGWB through dynamical friction (DF) on the inspiraling binary. Solving the Gross–Pitaevskii–Poisson equations numerically for halo masses M halo ∼ 10 13 M ⊙ , we compute self-consistent solitonic density profiles across a range of dimensionless self-coupling parameters λ ^ , covering both attractive and repulsive self-interactions. We find that soliton-induced DF imprints a characteristic suppression feature in the GW strain spectrum, whose frequency location and depth are sensitive to both the ULDM mass m and the coupling λ ^ . For SMBH masses M ∙ ∼ 10 8.5 M ⊙ and ULDM masses in the range m ∼ ( 3 × 10 − 22 − 2 × 10 − 21 ) eV , current PTA observations from NANOGrav, EPTA, and PPTA may place qualitative limits on the attractive and repulsive self-coupling of ULDM particles. ULDM masses exceeding ∼ 1.6 × 10 − 21 eV are independently excluded by soliton accretion-timescale constraints. These results demonstrate that the nanohertz GW spectrum provides a complementary probe of ULDM self-interactions, operating in a regime distinct from conventional particle-scattering or rotation-curve analyses.