AI-Assisted Interpretation of Children’s Drawings via Retrieval-Augmented Generation

This paper presents a retrieval-augmnted, CLIP-based pipline to support educators and psycologists in interpreting children’s drawings. The proposed system integrates vision-language captioning, case based retrival, and large language model generation, where CLIP produces descriptive captions, FAISS retrieves the top-k=3 semantically similar expert-annotated cases, and GPT-3.5 synthesizes a psychologically informed narrative report. Evaluation on 1,222 drawings with expert reference analyses considers six controlled variants (A1–A6) across two axes: (i) semantic similarity to expert interpretations (cosine similarity, BERTScore, ROUGE-L F1) and (ii) coverage and robustness (content-unit precision/recall/F1, ROUGE-L recall, new-information rate, and negation mismatch). In addition to the CLIP/BLIP ablations (A1–A4), we benchmark two image-only multimodal VLM baselines (GPT-4o and LLaVA; A5–A6) and conduct a small blinded expert rating study on 20 randomly selected drawings to contextualize trade-offs. Results show that retrieval-grounded configurations improve coverage compared to non-retrieval baselines and, in the CLIP-based comparison (A3 vs. A4), reduce polarity (negation) mismatches while maintaining competitive semantic similarity. Although GPT-4o attains the strongest similarity and content-unit scores overall, it incurs substantially higher latency and does not provide retrieval-based evidence trails; LLaVA shows mixed gains with notably lower content recall. The proposed CLIP-based variant achieves a practical balance of faithfulness and efficiency for deployment, and is explicitly positioned as decision support rather than diagnosis, surfacing salient cues and transparent evidence trails for expert oversight. Overall, this work demonstrates that retrieval grounding can make AI-assisted interpretation of children’s drawings more consistent, transparent, and scalable in educational and clinical contexts.