Image to sentiment pipelines for sustainable cities: a comparison of human and AI responses in urban environmental image analyses

Living within planetary boundaries while valuing society and nature requires a better understanding of citizens' perceptions of sustainability, allowing decision-makers to incorporate citizen sentiments and priorities into sustainable city planning (SDG 11). Citizen science qualitative data are increasingly accessible, and Artificial Intelligence-embedded pipelines are one approach to analyse such data and enable decision-makers to incorporate citizen sentiments and priorities into sustainable city planning (SDG 11). However, despite the often-reported high AI performance, its black-box nature can make the results difficult to interpret, evaluate and trust. In this study, we developed a pipeline for image-to-sentiment analysis, which enabled an effective comparison of AI and human responses. Human participants responded to a survey by assigning sentiment ratings to and describing a set of urban landscape images. For the same set of images, three vision-language models generated captions. Image captions were transformed into a sentiment value by AI models, allowing for a comparison of human responses and AI outputs through data science and statistical approaches. AI-generated captions and human descriptions largely overlap in vocabulary. When comparing the extracted sentiment, some AI algorithms performed more similarly to humans than others, regardless of algorithmic complexity. AI-human sentiment misalignment was lower for images humans described as neutral and negative, but larger for positive ones, potentially because of the more nuanced sentiment expressed by participants. Overall, effort is still required to enhance AI-human alignment. Further developments for city-wide applications will need to combine AI model strengths and human insights in a co-pilot framework, enriching sustainable city planning.