Student-Emotion Dataset Creation and Vision-Language Model Fine-Tuning for Classroom Engagement Analysis

This project targets emotion-aware understandingof classroom scenes by building a custom dataset from classroomimagery and fine-tuning a vision-language model on the resultingannotations. We propose a scalable pipeline that (i) detectsstudents, faces, and hands using an open-set grounding detector;(ii) associates detected parts to individual students; (iii) generatesglobal and region-level captions using an instruction-followingcaptioning model; (iv) assigns face-based emotion labels usingfacial analysis; and (v) stores outputs in a structured JSONformat including bounding boxes, captions, confidence scores, andtoken-span grounding. For modeling, we fine-tune a generativevision-language model (GIT-base-COCO) by constructing emotion-enhanced captions (e.g., “a bored student ...”) and training withlanguage-modeling loss. The results show that domain-specificcaption and emotion supervision can effectively adapt a pre-trained vision-language model to classroom engagement contexts.