Instability of Nagaoka state and quantum phase transition via kinetic frustration control

We investigate the Nagaoka-Thouless ferromagnetic instability in the strongly interacting t − t ′ Hubbard model by continuously breaking particle-hole symmetry on a tunable square-triangular lattice geometry. We use an analytic approach to show that the fully spin-polarized state becomes unstable to a metastable spin polaron when the kinetic frustration t ′ / t exceeds a critical, dimension-dependent value. Large-scale density matrix renormalization group simulations reveal a quantum phase transition from the Nagaoka ferromagnet to a spiral spin-density wave, which evolves continuously into the Haerter-Shastry antiferromagnet in the large-frustration limit. Remarkably, this transition remains robust at low but finite hole density, making it accessible in cold-atom and moiré Hubbard platforms under strong interactions. A variational analysis further captures the instability mechanism at finite hole density via frustration-induced magnon band deformation.