Quantification of Soil Organic Carbon by Shifted-Excitation Raman Difference Spectroscopy with Machine Learning and Common Mode Rejection

Soil organic carbon (SOC) is an important indicator of soil health and productivity in agricultural and natural ecosystems. Conventional measurement techniques, such as combustion analysis, yield accurate results but consume time and material resources. Raman spectroscopy combined with machine learning offers a rapid and ultimately field-deployable alternative. However, Raman measurements of soils face challenges due to matrix effects, such as fluorescence interference and light absorption by organic matter, which reduce the accuracy of machine-learning predictions. Here, we measure SOC in a data set of 400 American farm soils and apply machine learning to Shifted-Excitation Raman Difference Spectroscopy (SERDS) spectra, a fluorescence-free Raman scattering method. Our results show that for higher organic carbon concentrations, light absorption nonlinearly attenuates Raman spectroscopic features. This degrades the prediction accuracy of conventional machine-learning multivariate regression models for higher SOC levels, particularly when calibrated on data sets skewed to contain few high-SOC standards. A 200-sample synthetic data set of balanced mineral-organic mixtures serves to isolate the effects of light absorption and data set skew on Raman signal intensities. This work also compares two SERDS spectral preprocessing algorithms for soil analysis: Asymmetric Least Squares and Common-Mode Rejection. We find that Common-Mode Rejection and a nonlinear, tree-based machine learning model provide the most accurate results in the face of overwhelming fluorescence and nonlinear matrix effects inherent to soil samples.