Supplementary materials for the paper "Gemini Goes to Court: A Mixed-Methods Study of LLM Potential to Reduce Court Clerks' Workload"

Research Artifacts This repository contains an archive of the research artifacts for the paper "Gemini Goes to Court: A Mixed-Methods Study of LLM Potential to Reduce Court Clerks' Workload". Overview This repository contains materials from a study evaluating the performance and user experience of using the LLM (Gemini Pro) in supporting court clerks at drafting preparation documents from criminal case files. The study involved 15 court clerks (G1-G15) who processed two criminal case files under different conditions: without LLM support (Baseline, BSL) and with LLM support (Intervention, INT). Repository Structure 📁 criminal_files/ Contains the two criminal case files used in the study: - PR Criminal File.pdf - PR criminal case file - CVOM Criminal File.pdf - CVOM criminal case file 📁 data/ data.csv - contains the experimental data: Clerk identifiers (G1-G15) Which case the clerk considered in Baseline (BSL) and Intervention (INT) scenarios Duration of task execution in Baseline (BSL) and Intervention (INT) scenarios DeepEval Quantifier (DEQ) scores across four dimensions: PA: Prompt Alignment AR: Answer Relevancy CR: Contextual Recall HA: Hallucination Judge Assessment (JA) evaluation scores for: LC: Legally Correct? CO: Complete? AQ: Acceptable Quality? EI: Enough Information? Responses coded as Y (Yes), N (No), or M (Missing) 📁 sources/ Contains data analysis notebook and configuration: data_analysis.ipynb - Jupyter notebook with statistical analysis and visualizations pyproject.toml - Project dependencies and configuration uv.lock - Locked dependency versions Required Python version: 3.11-3.14 Dependencies: pandas (with excel, output-formatting, performance, and plot extras) scipy seaborn cliffs-delta Development dependencies: ipykernel 📁 supplementary/ Contains supplementary materials: criminal_files_description.pdf - Description of the criminal case files interview_protocols.pdf - Protocols used for conducting participant interviews prompt.pdf - LLM prompt used in this study translated into English Data Description Experimental Design Participants: 15 court clerks (G1-G15) Design: Counterbalanced crossover Conditions: Baseline (BSL): Standard workflow without the LLM assistance Intervention (INT): Workflow with the LLM assistance Case Files: PR and CVOM criminal cases Measurements Duration: Time spent on each case file (in seconds) DeepEval Quantifier (DEQ): Automated evaluation of LLM-generated outputs across four quality dimensions using the DeepEval framework (scale: 0.00-1.00) Judge Assessment (JA): Expert evaluation of document quality by a professional judge across four criteria (Y/N/M) Usage To reproduce the analysis: 1. Install dependencies using uv: bash cd sources/ uv sync 2. Run the analysis notebook: From command line: uv run jupyter notebook data_analysis.ipynb (install notebook dependency) In VS Code: Open data_analysis.ipynb and click "Run All" (you have to install VS Code extensions to work with Jupyter files) License This work is licensed under CC-BY 4.0.