ReaGP: integrating residual units and attention mechanisms in convolution neural network for genomic prediction

Abstract Background Various methods have been widely utilized to estimate the genomic breeding values (GEBVs) for genomic prediction. Traditional approaches often relied on the assumption of linear regression models, which struggle to effectively capture the nonlinear relationships between limited phenotypic data and high-dimensional genotypic data. Deep learning (DL) provided a powerful solution for addressing nonlinear problems. Herein, we proposed a novel deep learning method, named residual attention genomic prediction (ReaGP), which was characterized by two main features. It employed residual units to mitigate gradient instability and network degradation issues, while leveraging attention mechanisms to enhance the mining of critical feature information. Moreover, genomic data processed with frequency encoding was integrated into ReaGP to achieve a richer feature representation. Results When assessing the predictive accuracy across three animal datasets and two plant datasets covering 15 traits with varying heritabilities, ReaGP improved predictive performance by 14.41% and 7.78% over linear models specifically genomic best linear unbiased prediction (GBLUP) and BayesB, and by 34.41% and 10.09% over kernel methods namely support vector regression (SVR) and reproducing kernel Hilbert space (RKHS), respectively. ReaGP achieved a 4.35% enhancement on average compared to deep neural network genomic prediction (DNNGP). Furthermore, while ReaGP has more trainable parameters than DNNGP, it requires only half the number of floating-point operations. Conclusions We introduced a novel deep learning method for genomic prediction, which integrates residual units, attention mechanisms and frequency-encoded genomic data. Comprehensive evaluation on pig, dairy cow, Huaxi cattle, wheat and rice datasets demonstrated that ReaGP was a promising tool for most traits. Thus, ReaGP could be considered as an efficient deep learning method for genomic prediction in farm animals and crops. The source code in this study is available at https://github.com/LiJing5467/ReaGP .