Evaluating LLMs for Police Decision-Making: A Framework Based on Police Action Scenarios
Abstract
The use of Large Language Models (LLMs) in police opera- tions is growing, yet an evaluation framework tailored to po- lice operations remains absent. While LLM’s responses may not always be legally “incorrect”, their unverified use still can lead to severe issues such as unlawful arrests and improper evidence collection. To address this, we propose PAS (Po- lice Action Scenarios), a systematic framework covering the entire evaluation process. Applying this framework, we con- structed a novel QA dataset from over 8,000 official docu- ments and established key metrics validated through statis- tical analysis with police expert judgements. Experimental results show that commercial LLMs struggle with our new police-related tasks, particularly in providing fact-based rec- ommendations. This study highlights the necessity of an ex- pandable evaluation framework to ensure reliable AI-driven police operations. We release our data and prompt template.
Fields of Study & Subjects
Ethics and Social Impacts of AI
Computational and Text Analysis Methods
Topic Modeling
Publication details
- Journal / source
- Proceedings of the AAAI Conference on Artificial Intelligence
- Publisher
- Association for the Advancement of Artificial Intelligence (AAAI)
- Volume
- 40
- Issue
- 45
- Pages
- 38817–38825
- Publication date
- 14 March 2026
- Language
- EN