Bifurcation, multistability, and chaotic attractors in a predator-prey-mutualist system with nonlinear anti-predator defense

In this paper, a predator-prey-mutualist system with nonlinear protective anti-predator behavior is studied. Local stability, global stability, and bifurcation analysis are included. A rich variety of dynamical behaviors is shown by our study, including saddle-node bifurcation, transcritical bifurcation, Hopf bifurcation, degenerate Hopf bifurcation, and Bogadanov-Takens bifurcation. Numerical simulations further demonstrate the coexistence of multiple attractors, multi-stable limit cycles, and saddle-node bifurcation of limit cycles. Notably, this work presents, for the first time, the emergence of three distinct limit cycles and chaotic dynamics induced by nonlinear anti-predator effects. The influence of nonlinear anti-predator behavior on system dynamics is systematically examined. Specifically, the anti-predator rate and half-saturation constant serve as critical bifurcation parameters that regulate chaotic oscillations and promote stable coexistence of all species. Notably, an excessively high anti-predator rate may drive predator populations to extinction, while prey and mutualist populations asymptotically approach their respective carrying capacities. Under stable regime conditions, enhanced nonlinear anti-predator responses induce a monotonic decrease in predator biomass, concomitant with incremental increases in both prey and mutualist population densities. This study provides empirically grounded management frameworks to enhance biodiversity conservation.