Evaluation of the performance of multiple CMIP6 climate models against observational datasets (CHIRPS for precipitation and CHIRTS for temperature) over a specific region (Ganale sub-basin)

Project Description This project evaluates the performance of multiple CMIP6 climate models against observational datasets (CHIRPS for precipitation and CHIRTS for temperature) over a specific region (Ganale sub-basin). The evaluation involves calculating key metrics such as Bias, Mean Absolute Error (MAE), Root Mean Square Error (RMSE), and Pearson Correlation. A Composite Reliability Index (CRI) is then derived to rank the models based on their overall agreement with observations for both rainfall and temperature. The ultimate goal is to identify high-performing models suitable for multi-model ensemble (MME) generation, which can improve the reliability of future climate and hydrological assessments in the study region. Data Sources Observed Rainfall (CHIRPS): UCSB-CHG/CHIRPS/DAILY Observed Temperature (CHIRTS): UCSB-CHG/CHIRTS/DAILY CMIP6 Climate Models (Precipitation & Temperature): NASA/GDDP-CMIP6 Methodology Data Loading: Precipitation (pr) and temperature (tas) data from 22 CMIP6 models and corresponding observational datasets (CHIRPS pr, CHIRTS tmean) are loaded from Google Earth Engine for the period 1985-2014. Preprocessing: Daily precipitation is converted to monthly totals (mm), and daily minimum/maximum temperatures are averaged to monthly mean temperature (Celsius). Spatial and Temporal Alignment: All datasets are resampled to a common spatial resolution (0.05 degrees) and aggregated to monthly values. Performance Metrics: For each CMIP6 model, the following metrics are calculated against the respective observational dataset: Bias (mean difference) Mean Absolute Error (MAE) Root Mean Square Error (RMSE) Pearson Correlation Coefficient Composite Reliability Index (CRI): A CRI is calculated for both rainfall and temperature models based on the rankings of the four performance metrics. A higher CRI indicates better model performance. Model Selection: Models are ranked by their combined CRI (average of rainfall and temperature CRIs) to identify the top-performing models for multi-model ensemble generation. Key Findings The notebook presents detailed performance metrics and CRI scores for 22 CMIP6 models for both rainfall and temperature. Heatmaps and bar plots are used to visualize model performance and ranking. Specific models are identified as high-performing based on their CRI scores, demonstrating strong agreement with observed data. The analysis provides a data-driven approach for selecting suitable GCMs for regional climate impact studies. Setup and Usage T his notebook can be run in Google Colab. Ensure you have access to Google Earth Engine. Installation pip install geemap xee Authentication Authenticate with Google Earth Engine using ee.Authenticate() and ee.Initialize(project='your-ee-project-id'). Run the Notebook Execute cells sequentially to replicate the analysis. The final combined CRI results and visualizations will be generated. Outputs monthly_stack_data.csv: CSV file containing the xarray Dataset converted to a pandas DataFrame. rmse_map_prX.png: Spatial RMSE maps for each precipitation model. DataFrames (df_rainfall_cri, df_temperature_cri, df_combined_cri_sorted) containing model performance metrics and CRI scores. References Xu et al. (2024). [Insert actual reference if available] requirements.txt %%writefile requirements.txt ee geemap xarray numpy pandas xee matplotlib cartopy seaborn General Report: Climate Model Intercomparison and Evaluation for Rainfall and Temperature Abstract This study presents a comprehensive evaluation of 22 Coupled Model Intercomparison Project Phase 6 (CMIP6) General Circulation Models (GCMs) for their ability to simulate historical rainfall and temperature in the Ganale sub-basin region. Utilizing observational datasets from CHIRPS (for precipitation) and CHIRTS (for temperature) spanning 1985-2014, we assessed model performance using key metrics: Bias, Mean Absolute Error (MAE), Root Mean Square Error (RMSE), and Pearson Correlation. To synthesize these metrics, a Composite Reliability Index (CRI) was developed for both rainfall and temperature, allowing for a robust ranking of models. The findings demonstrate varying degrees of model skill, with specific models consistently outperforming others across both variables. This systematic evaluation provides a data-driven framework for selecting optimal GCMs, culminating in the identification of six high-performing models (INM-CM4-8, CanESM5, KACE-1-0-G, HadGEM3-GC31-LL, INM-CM5-0, and GISS-E2-1-G) suitable for multi-model ensemble (MME) generation to enhance the reliability of future climate and hydrological projections in the study area. Introduction Accurate climate projections are crucial for water resource management, agricultural planning, and disaster preparedness, particularly in regions vulnerable to climate change impacts. General Circulation Models (GCMs) are fundamental tools for climate projection, yet their performance varies significantly across models and geographical regions. This study focuses on evaluating CMIP6 GCMs over the Ganale sub-basin to identify those that best represent historical climate conditions, thereby increasing confidence in their future projections. The objective is to provide a robust methodology for GCM selection, essential for regional climate impact assessments. Data and Methods 2.1. Observational Data Rainfall: Climate Hazards Group InfraRed Precipitation with Station data (CHIRPS) daily precipitation, aggregated to monthly totals (mm). Period: 1985-2014. Temperature: Climate Hazards Group IR Temperature (CHIRTS) daily minimum and maximum temperatures, averaged to monthly mean temperature (Celsius). Period: 1985-2014. 2.2. CMIP6 Model Data Twenty-two CMIP6 GCMs from the NASA/GDDP-CMIP6 dataset were selected for historical simulations of precipitation (pr) and near-surface air temperature (tas). Daily model outputs were processed to match the monthly temporal resolution of observational data. Precipitation was converted from kg/m²/s to monthly accumulated millimeters, and temperature from Kelvin to Celsius. 2.3. Spatial and Temporal Processing All datasets (observational and model) were filtered for the common period (1985-2014) and spatially clipped to the Ganale sub-basin region. A consistent spatial resolution of 0.05 degrees was ensured for all data products. 2.4. Performance Metrics For each CMIP6 model, against its respective observational counterpart (CHIRPS for pr, CHIRTS for tas), the following statistical metrics were computed: Bias: Mean difference (Model - Observation). Mean Absolute Error (MAE): Average of the absolute differences. Root Mean Square Error (RMSE): A measure of the magnitude of the errors. Pearson Correlation Coefficient: Measures the linear relationship between model and observation. 2.5. Composite Reliability Index (CRI) A Composite Reliability Index (CRI) was developed to provide an aggregated measure of model performance. For each variable (rainfall and temperature), models were ranked based on each of the four metrics (Bias, MAE, RMSE ranked ascending; Correlation ranked descending). The sum of ranks was used to compute CRI using the formula: CRI=1−(Total Rank/(n×m)), where n is the number of metrics (4) and m is the number of models (22). A higher CRI value indicates superior model performance. 2.6. Multi-Model Ensemble (MME) Selection To identify a robust subset of models for MME generation, the CRI scores for rainfall and temperature were averaged to derive a 'Combined CRI'. Models with the highest Combined CRI were selected as the best performers. Results 3.1. Rainfall Model Evaluation The evaluation of rainfall models revealed significant differences in performance. FGOALS-g3 (pr10) consistently ranked highest across various metrics, demonstrating the lowest RMSE and MAE, and highest correlation. This was reflected in its superior CRI score for rainfall. Spatial RMSE maps for each model highlighted regional biases and errors, providing insights into areas where models struggle. 3.2. Temperature Model Evaluation For temperature, models generally showed better agreement with observations compared to rainfall, characterized by lower RMSE values. INM-CM4-8 (tas15) emerged as the top-performing model for temperature, exhibiting the lowest Bias, MAE, and RMSE, and highest correlation. This model achieved the highest CRI score for temperature. 3.3. Combined Reliability Index and Model Selection The Combined CRI, averaging the rainfall and temperature CRIs, provided an overall ranking of the GCMs. The top six models identified were INM-CM4-8, CanESM5, KACE-1-0-G, HadGEM3-GC31-LL, INM-CM5-0, and GISS-E2-1-G. These models demonstrated strong agreement with both CHIRPS and CHIRTS datasets, making them ideal candidates for forming a multi-model ensemble to improve the reliability of future climate assessments in the Ganale sub-basin. Discussion The methodology presented offers a quantitative and transparent approach to GCM evaluation and selection, moving beyond reliance on single metrics. The divergence in model performance underscores the importance of regional evaluation before employing GCMs for impact studies. The identified top-performing models can be used to construct a regional multi-model ensemble, which has been shown to reduce uncertainties and improve the robustness of climate change projections. Future work will involve utilizing these selected models for downscaling and projecting future climate scenarios for the Ganale sub-basin. Conclusion This study successfully evaluated 22 CMIP6 GCMs against observational rainfall and temperature data for the Ganale sub-basin. A Composite Reliability Index (CRI) proved effective in ranking models based on multiple performance metrics. The selection of six high-performing models—INM-CM4-8, CanESM5, KACE-1-0-G, HadGEM3-GC31-LL, INM-CM5-0, and GISS-E2-1-G—provides a robust foundation for future climate and hydrological impact studies in the