Can LLMs Truly Embody Human Personality? Analyzing AI and Human Behavior Alignment in Dispute Resolution

Large language models (LLMs) are increasingly used to simulate human behavior in high-stakes social settings such as legal mediation, negotiation, and dispute resolution. However, it remains unclear if the LLM-based models simulating human behavior accurately represent the underlying psychological mechanisms. Human personality, for instance, may shape how individuals navigate social interactions, including strategic choices and behaviors in emotionally charged interactions. This raises a critical question: Can LLMs, when prompted with personality traits, reproduce personality-driven differences in human conflict behavior? To explore this, we introduce an evaluation framework that compares human-human and LLM-LLM behaviors in dispute resolution dialogues concerning Big Five Inventory personality traits. Our contributions include introducing a novel methodology to create a dataset of human and LLM dialogs with matched scenarios and personality traits and a set of interpretable metrics capturing strategic and conflict outcome dynamics. We analyze three recent closed-source LLMs and show significant divergences in how personality manifests in conflict across different LLMs compared to human data, challenging the assumption that personality-prompted agents can serve as reliable behavioral proxies in socially impactful applications. Our work highlights the need for psychological grounding and rigorous validation in AI simulations before real-world use.