A Progressive Evaluation Framework for Multicultural Analysis of Story Visualization

Recent advancements in text-to-image generative models have improved narrative consistency in story visualization.However, current story visualization models often overlook cultural dimensions, resulting in visuals that lack cultural fidelity.In this study, we present a progressive evaluation framework for story visualization.We validate this framework on current text-to-image models across three languages (English, Hindi, and Chinese) on two datasets (VIST and FlintstonesSV).The proposed framework introduces three levels of cultural analysis as evaluation rubrics: 1) Basic Cultural Criteria, 2) Cultural Dimension Guidance, and 3) Cultural Examples Grounding.We evaluate story visualization by use of a novel MLLM-as-Jury approach across all three rubrics and a small-scale human evaluation only on the third rubric.We implement an MLLM-as-jury approach by aggregating scores from three different families of MLLM-as-Judge models.In our experiments, real-world stories generally receive higher cultural appropriateness scores than animated ones, with English tending to score higher than Hindi and Chinese across the evaluated models.Some examples also exhibited culturally inconsistent or stereotypical elements noted by annotators.The proposed progressive evaluation framework has therefore been shown to provide early insights into cultural misalignments in story visualization.Code for this work is made available on https://github.com/janak11111/ Cultural_Eval_For_StoryViz