Search for resonances decaying to an anomalous jet and a Higgs boson in proton–proton collisions at $$\sqrt{s}=13\,\text {Te}\hspace{-.08em}\text {V} $$

Abstract This paper presents a search for new physics through the process where a massive particle, X, decays into a Higgs boson and a second particle, Y. The Higgs boson subsequently decays into a bottom quark–antiquark pair, which is reconstructed as a single large-radius jet. The decay products of Yare also assumed to produce a single large-radius jet. The identification of the Yparticle is enhanced by computing the anomaly score of its candidate jet using an autoencoder, which measures deviations from typical quark- or gluon-induced jets. This allows a simultaneous search for multiple Ydecay scenarios within a single analysis. In the main benchmark process, Yis a scalar particle that decays into a Wboson pair. Two other scalar Ydecay processes are also considered as benchmarks: decays to a light quark–antiquark pair, and decays to a top quark–antiquark pair. A fourth benchmark process considers Yas a hadronically decaying top quark, arising from the decay of a vector-like quark into a top quark and a Higgs boson. Data recorded by the CMS experiment at a center-of-mass energy of 13 $$\,\text {Te}\hspace{-.08em}\text {V}$$ Te V in 2016–2018, corresponding to an integrated luminosity of 138 $$\,\text {fb}^{-1}$$ fb - 1 , are analyzed. The search covers Xmasses between 1.4 and 3.0 $$\,\text {Te}\hspace{-.08em}\text {V}$$ Te V and Ymasses between 90 and 400 $$\,\text {Ge}\hspace{-.08em}\text {V}$$ Ge V , with all simulated signals produced in the narrow-width approximation. No significant excess above the standard model background expectation is observed. The most stringent upper limits to date are placed on benchmark signal cross sections for various masses of X and Y particles