UniCrop: A Universal, Multi-Source Data Engineering Pipeline for Scalable Crop Yield Prediction

Accurate crop yield prediction increasingly relies on diverse data streams, including satellite observations, meteorological reanalysis, soil composition, and topographic information. However, despite rapid advances in machine learning, most existing approaches remain crop-or region-specific and require substantial bespoke data engineering efforts. This limits scalability, reproducibility, and operational deployment. This study introduces UniCrop, a universal and reusable data pipeline designed to automate the acquisition, cleaning, harmonisation, and feature engineering of multi-source environmental data for crop yield prediction. For any given location, crop type, and temporal window, UniCrop automatically retrieves, harmonises, and engineers over 200 environmental variables from heterogeneous satellite, climate, soil, and topographic sources (Sentinel-1/2, MODIS, ERA5-Land, NASA POWER, SoilGrids, and SRTM), reducing them to a compact, analysis-ready feature set utilising a structured feature reduction workflow with minimum redundancy maximum relevance (mRMR). To validate the pipeline, UniCrop was applied to a rice yield dataset comprising 557 field observations. Using only the selected 15 features, four baseline machine-learning models (LightGBM, Random Forest, Support Vector Regression, and ElasticNet) were trained using rigorous crossvalidation. LightGBM achieved the best single-model performance (RMSE = 465.1 kg/ha, R 2 = 0.6576), while a constrained ensemble of all baselines further improved accuracy (RMSE = 463.2 kg/ha, R 2 = 0.6604). SHAP analysis confirmed agronomically plausible relationships and demonstrated how UniCrop leverages multi-modal predictors. UniCrop contributes a scalable and transparent data-engineering framework that addresses the primary bottleneck in operational crop yield modelling: the preparation of consistent and harmonised multi-source data. By decoupling data specification from implementation and supporting any crop, region, and time frame through simple configuration updates, UniCrop provides a practical foundation for transferable, high-quality agricultural analytics at scale. The code and implementation documentation are shared in https://github.com/CoDIS-Lab/UniCrop.

💡 Research Summary

The paper introduces UniCrop, a universal, reusable data‑engineering pipeline designed to streamline the acquisition, cleaning, harmonisation, and feature engineering of heterogeneous environmental data for crop‑yield prediction. Recognising that the most time‑consuming bottleneck in modern yield modelling is the preparation of consistent multi‑source inputs, the authors built a system that automatically pulls data from six major repositories—Sentinel‑1/2 SAR and optical imagery, MODIS vegetation indices, ERA5‑Land reanalysis climate fields, NASA POWER climate variables, SoilGrids soil properties, and SRTM topography. For any user‑specified location, crop type, and temporal window, UniCrop retrieves the raw observations, resamples them to a common spatial (30 m) and temporal resolution, and applies a robust cleaning routine: missing values are imputed using neighbouring observations, outliers are detected via inter‑quartile range filters and corrected with multivariate regression. The pipeline then generates more than 200 raw variables covering temperature, precipitation, radiation, soil texture, organic carbon, pH, moisture, elevation, slope, aspect, and a suite of NDVI/EVI time‑series metrics.

To reduce dimensionality while preserving predictive power, the authors employ a structured feature‑selection workflow based on minimum Redundancy Maximum Relevance (mRMR). This method simultaneously minimises inter‑feature correlation and maximises mutual information with the target yield, yielding a compact set of 15 highly informative predictors. The selection process is embedded within a 5‑fold cross‑validation loop to ensure stability and avoid over‑fitting.

The pipeline’s efficacy is demonstrated on a rice‑yield dataset comprising 557 field observations. Four baseline machine‑learning models—LightGBM, Random Forest, Support Vector Regression, and ElasticNet—are trained on the 15 selected features using rigorous cross‑validation. LightGBM achieves the best single‑model performance (RMSE = 465.1 kg ha⁻¹, R² = 0.6576). A constrained ensemble that averages the predictions of all four baselines yields a modest improvement (RMSE = 463.2 kg ha⁻¹, R² = 0.6604). To interpret model behaviour, SHAP (Shapley Additive Explanations) values are computed, revealing agronomically plausible relationships: mean seasonal temperature, precipitation variability, soil organic carbon, and NDVI dynamics emerge as the strongest drivers of rice yield, aligning with established agronomy literature.

From an engineering perspective, UniCrop is implemented as a modular Python package with a configuration‑driven interface. Users can switch crops, regions, or time periods simply by editing a YAML file, without altering code. The entire codebase, along with detailed documentation, Docker images, and continuous‑integration pipelines, is openly released on GitHub (https://github.com/CoDIS-Lab/UniCrop). This openness ensures reproducibility, facilitates community contributions, and enables rapid deployment in operational settings such as early‑warning systems or precision‑agriculture platforms.

In summary, UniCrop addresses the primary obstacle to scalable, transferable yield modelling—data preparation—by automating the end‑to‑end workflow from raw satellite and reanalysis products to an analysis‑ready feature matrix. The pipeline’s flexibility, transparency, and demonstrated predictive performance make it a valuable foundation for future research across diverse crops, geographies, and temporal scales, and it paves the way for integrating real‑time data streams into operational agricultural decision‑support tools.