Let’s Decipher the Origin: Towards Multimodal Aspect-Based Explainable Complaints in Finance

Financial grievances on social media increasingly feature multimodal content, combining text and images to express complex concerns. Previous text-only approaches often struggle to capture mixed emotional phenomena within posts, such as praise for credit card services alongside dissatisfaction with branch wait times. Prior research has relied on binary classifications, overlooking root aspect-specific complaints (e.g., categorizing card services as noncomplaint while branch services are complaint-related). To address the scarcity of multimodal data resources for this task, we developed the financial multimodal complaint (FMC) dataset, comprising over 6,000 tweets across nine financial domains, capturing cross-modal complaint dynamics. To demonstrate the practical use of the FMC dataset, we designed F-ACE, a multimodal, aspect-based, explainable complaint identification model that integrates BLIP-2, Sentence Transformer, and ResNet-152 via an intermodal fusion (IMF) framework, followed by a BART-Large-based causal extractor for nuanced aspect classification. Furthermore, we evaluate the proposed model’s comprehension and adaptability through the utilization of various domain-specific and cross-domain datasets alongside state-of-the-art models. Our contribution sets a preferable alternative for advancing research in aspect-aware, multimodal complaint analysis, enabling a deeper understanding of financial grievances in the digital age.