Mitigating nonlinear transduction noise in high-cooperativity cavity optomechanics

Coupling mechanical motion to an optical resonator enables displacement measurements approaching the standard quantum limit (SQL). However, increasing the optomechanical coupling strength will inevitably lead to probing of the nonlinear response of the optical resonator. Thermal intermodulation noise (TIN) arising from the nonlinear mixing of thermomechanical motion can further increase the imprecision well above the SQL and has hitherto been canceled up to second order of nonlinearity via operation at the "magic detuning". In this work, we record the output of a membrane-in-the-middle microcavity system operating at room temperature and achieving high cooperativity, $C>n_\text{th}$, and apply a nonlinear transform that removes all orders of TIN, improving the mechanical signal-to-noise ratio by nearly 10 dB. Our results can be applied to experiments affected by third-order TIN, which we expect to be the dominating intrinsic source of noise in high-cooperativity room-temperature cavity optomechanical systems.