SMART: Structured multimodal agricultural retrieval-augmented transformer for plant disease diagnosis

Real-world plant disease diagnosis requires robustness to long-tailed class distributions and variable input quality, yet prevailing multimodal approaches do not explicitly address these challenges. This study presents SMART, a framework that integrates structured linguistic guidance with retrieval-augmented inference. Evaluation of 37,586 images across 48 disease categories yields three findings. First, structured captions outperform LLM-generated narratives, with the vision-only baseline exceeding all LLM approaches on the 20-class pilot, demonstrating that semantic precision, not descriptive complexity, drives effective vision-language alignment. Second, we distinguish between training-time and inference-time challenges: long-tailed degradation is mitigated through asymmetric loss design, while input quality variability is addressed through a dual-pathway mechanism in which semantic anchoring constrains predictions under related errors, and the retrieval-based safety net ensures bounded degradation under noisy inputs. Third, frequency-stratified analysis reveals a division of labor: semantic anchoring benefits high-frequency classes, while the retrieval-based safety net preferentially compensates rare classes. These complementary mechanisms support a human-in-the-loop workflow that transforms single-pass diagnosis into collaborative verification. SMART achieves F1 = 0.9767, surpassing fine-tuned CNNs (best F1 = 0.9118) and 22 multimodal foundation models (best F1 = 0.8904). The primary contributions of this work are: (i) a systematic benchmark of caption form-structured versus LLM-generated as an experimental variable in agricultural vision-language alignment; (ii) a factorial decomposition that quantitatively isolates the independent and complementary roles of semantic anchoring and retrieval-based compensation across frequency strata; and (iii) a statistically validated, human-in-the-loop diagnostic framework applicable to large-scale, imbalanced disease taxonomies.