Leveraging the joint site frequency spectrum to detect genomic regions of early divergence

Abstract Detecting genomic loci underlying local adaptation is a central goal in population genetics. Such loci have been discovered using a variety of approaches, but the statistical performance of these approaches can depend substantially on the frequencies of the alleles underlying phenotypic adaptation. In particular, multiple evolutionary modeling studies have shown that rare alleles of large effect can make substantial contributions to phenotypic variance and potentially adaptive divergence, but most selection inference methods are not sensitive to rare alleles. We used simulations of evolutionary divergence to compare commonly-used F ST scans to a likelihood-based approach that interrogates the whole frequency spectrum. We found that the likelihood-based approach outperforms F ST in simulations in which low frequency alleles play a substantial role in driving trait divergence. We applied both approaches to genomic data from Ostrinia nubilalis (the European Corn Borer), a species in which population-specific variation in circannual rhythms and mate preference phenotypes have driven recent reproductive isolation. The likelihood-based approach recovers previously discovered genomic loci and finds several new candidate regions that may be relevant for reproductive isolation between populations. Our findings demonstrate how the two-dimensional frequency spectrum may help to identify loci contributing to reproductive isolation in contexts when commonly used methods are less powerful. Article summary Identifying genomic loci underlying phenotypic divergence is a fundamental challenge in population genetics. Some commonly used approaches, like F ST scans, perform poorly when rare alleles drive phenotypic differentiation, a scenario that arises under some polygenic trait architectures. We used forward simulations of selection under different trait architectures to compare F ST scans to a likelihood-based method, and found that the likelihood-based approach is more powerful when low-frequency alleles contribute to trait divergence. Applying both methods to the European Corn Borer moth, we recovered known loci and identified new candidate divergent regions, demonstrating the utility of likelihood-based approaches in population genomics.