Probing the content of semantic representations in body-selective regions

Recent advances in neural networks trained on natural language have revealed that category-selective regions encode complex semantics and contextual information of natural scenes in addition to object categories. However, the limited interpretability of embeddings derived from these models complicates the characterization of the aspects of natural scenes that contribute to such semantic representations. Here we addressed this question by developing an analysis of the relationship between object co-occurrence in large-scale natural scene captions and corresponding fMRI responses predicted by caption-based encoding models, inspired by the fact that the joint presence of multiple objects generally shapes the overall content of a scene. We performed this analysis on the extrastriate body area (EBA), which responds strongly to human body parts. We found that human bodies co-occurring with sports-related objects drive the strongest predicted responses in the EBA among all image categories, whereas those co-occurring with vehicles or accessories elicit strong but weaker predicted responses. The findings from the co-occurrence analysis helped identify three key body-related features that contribute to the semantic representation in the EBA and fusiform body area: human body motion speed implied in static images of natural scenes is a primary contributor, and the number of people and body size are secondary contributors. Our framework, integrating object co-occurrence with caption-based encoding models, offers an interpretable approach for understanding high-level visual representations underlying natural scene perception.