A posteriori unimodality for LLM bias detection

Recent studies show that biases exhibited by Large Language Models are not random, but the result of systematic and interpretable socio-demographic annotator perspectives, often reflecting majority-group viewpoints. In this work, we investigate whether \acp{LLM} reproduce similar polarization and subgroup-specific alignment patterns to those observed in human annotations. Firstly, we examined whether \acp{LLM} can become polarized in the same way as humans. Our analysis shows that six widely used \acp{LLM} (ChatGPT, DeepSeek, Gemini, Mistral, Llama, Grok) tend to align more strongly with the judgement patterns of female, white, and left-leaning subgroups when annotating toxic content, whereas humans themselves appear to be more strongly polarized along ideological attributes rather than hereditary characteristics. Then, we introduced a framework that examines how model behaviour changes when prompts explicitly ask the model to adopt demographic or ideological identities. Instead of aiming to reduce bias, this approach intentionally exposes language models to the same social viewpoints that drive polarization in the human data. This allows us to check whether model alignment with human groups reflects repetitive internal patterns or bias that emerges only when identity adoption happens at the prompt level. Finally, we extend the A Posteriori Unimodality framework, originally used to attribute polarization to specific dimensions—to support an extensive set of attributes. This subset includes ideological dimensions (e.g., politics and traditionalism) and ordinal attributes. For those, we introduce a threshold-based analysis that allows polarization to be identified across ordered subgroups, which in this study applies to age. This extension preserves the dimension-based nature of A Posteriori Unimodality while enabling finer-grained subgroup analysis and direct comparison with model behaviour. \\\textcolor{red}{\textbf{Disclaimer:} This thesis contains examples of toxic, offensive, and unsafe language collected from real-world datasets, included exclusively for research purposes and not intended to promote or endorse such content.}