Only Subsets Matters: The Effect of Dual Fairness Constraints in Speech Emotion Recognition

Speech Emotion Recognition (SER) systems are increasingly deployed in voice-centric applications, yet often suffer from fairness concerns due to speaker-induced variability. In particular, speaker-gender bias can cause systematic disparities in performance across demographic groups (group fairness), while emotionally expressive or acoustically unique speakers may be treated inconsistently despite similar input (individual fairness). Although prior work suggests that group and individual fairness objectives may be inherently incompatible, our proposed two-stage debiasing framework aims to address both: an in-processing approach first mitigates speaker-gender bias (group fairness), followed by a post-processing calibration step that improves consistency across similar instances (individual fairness). While most samples benefit from this dual intervention, we identify a smallsubsetof speech samples that remain difficult to classify fairly. This work focuses on systematically analyzing these fairness-ambiguous samples to understand what makes them challenging. We examine this question from two perspectives: emotion perception and acoustic expressivity. Our analyses on thesesubsetsindicate that: (1) exhibit extreme or atypical emotional ratings, (2) show high acoustic variability, and (3) tend to come disproportionately from specific individuals. These findings suggest that some speakers inherently present greater challenges to fairness optimization, due to the uniqueness of their emotional or acoustic expression. By characterizing thesesubsets, our work contributes to a deeper understanding of fairness conflicts in SER and offers new directions for developing more robust and inclusive emotion recognition systems.