How to Choose a Fairness Measure: A Decision‐Making Workflow for Auditors

ABSTRACT Recent developments in Artificial Intelligence (AI) have greatly benefited society but also introduced considerable risks. Among these risks is the potential for AI systems to produce discriminatory outcomes against specific social groups. In response, benchmark regulations such as the EU AI Act have been created, requiring AI systems to be fair and tasking auditors with ensuring their compliance. In practice, AI audits rely on fairness measures to perform an AI audit; however, two major challenges persist: selecting the most appropriate measure among the many available and defining fairness itself. To address these challenges, we developed a decision‐making workflow that guides auditors in identifying the most suitable fairness measure and, by extension, the corresponding definition of fairness. To facilitate its practical application, the workflow has been integrated into the open‐source program JASP for Audit, and its functionality is demonstrated through three illustrative case studies: the COMPAS recidivism case, the DUO case and the German Credit Risk case.