Dual-branch CNN-based fusion of computer vision and near-infrared spectroscopy for quantitative prediction: A case of black tea processing

• A dual-branch CNN-based fusion framework was proposed for quantitative analysis. • CV outperformed NIRS in the prediction accuracy of black tea components. • Mid fusion of features extracted by CNN achieved the highest predictive accuracy. • S-G-SNV/MSC showed advantages in reducing noise caused by tea leaf particles. • It’s an in-situ low-cost solution for quality monitoring during tea processing. The captivating flavor and potential health benefits of black tea are developed during its unique processing stages. At present, the assessment of black tea quality relies on the sensory evaluation of tea masters. However, there is a lack of quality monitoring methods suitable for industrial-scale production of black tea. This study presents a convolutional neural network (CNN)-based multimodal fusion approach integrating near-infrared spectroscopy (NIRS) and computer vision (CV) for the non-destructive quantification of components in black tea processing, including moisture, polyphenols (including catechins), and chlorophyll. A total of 600 tea samples were collected from four batches and six processing stages, with standard chemical methods used as reference measurements. Spectral preprocessing was applied, and traditional machine learning models (PLSR, SVR) were compared against deep learning architectures (1DCNN, 2DCNN). The results showed that CNN models significantly outperformed traditional machine learning. The image-based 2DCNN model outperformed the NIRS-based 1DCNN. Furthermore, the dual-branch CNN-based fusion of NIRS and CV achieved higher predictive accuracy, yielding the highest residual predictive deviation (RPD) values for the prediction of moisture (11.0774), polyphenols (4.1432), catechins (3.3912), and chlorophyll (7.8479). Overall, this study provides a rapid, cost-effective and on-site deployable solution for quality monitoring during black tea processing, facilitating targeted control of tea quality.