Investigating the Impact of Combined Spectral and Prosodic Features on Speech Emotion Recognition

Emotions play a fundamental role in human cognition, behavior, and social interaction, making automatic recognition a key topic in affective computing. Many existing approaches to recognizing emotions from speech rely heavily on Mel-Frequency Cepstral Coefficients (MFCCs), which capture short-term spectral features but insufficiently represent prosody, long-term dynamics, and tonal nuances that are critical for accurate classification. This study presents an interpretable and computationally efficient framework for recognizing emotions from speech by employing a compact set of fourteen spectral and prosodic acoustic features, including pitch, shimmer, jitter, loudness, harmonic-to-noise ratio (HNR), and measures of temporal variation. Using tree-based ensemble methods, the proposed system achieved its best performance with the XGBoost classifier, reaching an accuracy of 96.79% on the Toronto Emotional Speech Set (TESS). Statistical validation using the Kruskal–Wallis test and effect size analysis revealed that HNR, mean pitch, and shimmer were the most discriminative predictors of emotional state, thereby providing transparency into the classification process. The system also demonstrated real-time capability with inference times between 201 and 231 milliseconds, confirming that accurate, efficient, and interpretable speech emotion recognition can be achieved without relying solely on deep learning models.