A Metadata Driven Approach to Quantify and Overcome Data Fragmentation in Crop Models

Data such as climate, soil, irrigation, and remotesensing are scattered across multiple portals and diversified formats, thus leading to poor generalisation in robust agricultural machine learning models in India. Without a unified data discovery mechanism or platform, it becomes difficult to reproduce models and reduces their reliability. As a solution to this issue, we proposed a system, namely AgriSangam, a collaborative metadata-driven platform designed to organise agricultural datasets. In contrast to other repositories, our platform does not automatically ingest data, which enables the researcher to investigate the suitability of the data before it is integrated. To explain the platform need, a validation of such integration, we performed a controlled evaluation comparing identical XGBoost models trained on isolated single-domain sources (e.g., climateonly) with crop as a baseline against those trained on integrated datasets. We have implemented the exact protocols to prevent the spatiotemporal leakage, employing Out-of-Time (OOT) forecasting on 2016-2017 and a 5-fold Spatial Cross-Validation (CV) on held-out districts. Single-domain trained models had moderate predictive power ($R$-squared values between 0.66 and 0.69) that was primarily due to crop-type differences instead of generalisation. Conversely, the integrated models showed clear improvement in predictive accuracy with (OOT R-squared value is 0.744 and MAE is 0.442). Furthermore, spatial cross-validation demonstrated robust generalisation to unseen regions ($R$-squared of 0.789 with standard deviation of 0.026). Lastly, by applying SHAP-based analysis, we found that a smaller set of 49 features could retain the predictive power of (OOT R-squared of 0.747). The major problem identified from these findings states that data fragmentation is the primary bottleneck in agricultural AI. Thus validating the necessity of metadata-driven platforms like AgriSangam to facilitate systematic, multi-modal integration for real-world research.