Benchmarking Zero-Shot Multimodal Reasoning Models for Rooftop Photovoltaic Potential: A Consensus-Based Evaluation Framework

We study whether a zero-shot Multimodal Reasoning Model (MRM; a promptable visionlanguage foundation model) can produce usable rooftop annotations for photovoltaic (PV) assessment from aerial orthophotos. We evaluate a commercial MRM (Gemini 2.5 Pro) on 5,359 orthoimage patches and compare it against 7,402 human annotation sets. To address annotator variability, we propose a consensus-based evaluation protocol that clusters pooled human bounding boxes into an inclusive (union-based) human reference ("any-human-acceptable"). Unlike traditional supervised detectors, MRMs can integrate multimodal cues and context (e.g., semantic disambiguation and spatial relations) to interpret complex roof morphologies and to resolve ambiguous obstructions. Because provider interfaces typically expose only final outputs, we evaluate model behavior indirectly via the correctness and consistency of the structured annotations produced. In our primary evaluation setting, the MRM achieves roof-level agreement comparable to human consistency (consensus F1 = 0.684 vs. human leave-one-out F1 = 0.701), but underperforms substantially on rooftop obstructions (consensus F1 = 0.491 vs. human leave-one-out F1 = 0.584). We further show that obstruction performance is strongly scale-and class-dependent, with a salient failure mode in which the MRM treats large solar-panel arrays as roof surface rather than obstructions. These error patterns emphasize the need for contextual disambiguation in obstruction detection (e.g., distinguishing vents or skylights from shadows, texture, or rooftop clutter). To facilitate comparison against conventional supervised learning, we provide a minimal YOLOv8 baseline implementation and a reproducible dataset split.