Audio-Visual-Textual Fusion for Big Five Personality Prediction Using Deep Feature Extraction and Multimodal Transformers

This study presents a data-efficient, interpretable framework for personality trait prediction using multimodal behavioral cues. Leveraging both handcrafted and deep features from video, audio, and text, and integrating them through a Multimodal Transformer with cross-modal attention, our approach captures rich intra- and inter-modal dependencies critical to accurate personality inference. To enhance interpretability, we introduced a Gaussian Mixture Model (GMM)-based post-processing step that transforms continuous regression outputs into discrete personality categories—Low, Medium, and High. This conversion enables intuitive and human-readable predictions, making the system suitable for real-world applications such as recruitment, adaptive education, and behavioral monitoring. Experimental results show that fusing modalities with feature diversity and cross-modal attention yields robust personality prediction performance even with limited training data.