Standardised Annotation Frameworks for Geoscience Benchmark Datasets Using Text- and Image-Based Assessments

This work presents a structured human annotation framework for the development of high-quality geoscience benchmark datasets used in research and model evaluation. The methodology is designed to support accurate, standardised categorisation and response generation for two primary assessment formats: text-based geoscience questions and image-based questions incorporating maps, diagrams, and Earth observation imagery. The framework emphasises analytical consistency, scientific validity, and adherence to predefined annotation standards. Quality control procedures, including self-review and decision documentation, are incorporated to reduce ambiguity and improve reproducibility across annotated datasets. The approach is suitable for evaluation-oriented tasks requiring precise Earth science reasoning and reliable human-in-the-loop validation.