PyCFRL: A Python library for counterfactually fair offline reinforcement learning via sequential data preprocessing

Reinforcement learning (RL) aims to learn and evaluate a sequential decision rule, often referred to as a "policy", that maximizes expected discounted cumulative rewards to optimize the population-level benefit in an environment across possibly infinitely many time steps.RL has gained popularity in fields such as healthcare, banking, autonomous driving, and, more recently, large language model fine-tuning.However, the sequential decisions made by an RL algorithm, while optimized to maximize overall population benefits, may disadvantage certain individuals who are in minority or socioeconomically disadvantaged groups.A fairness-unaware RL algorithm learns an optimal policy that makes decisions based on the observed state variables.However, if certain values of the sensitive attribute influence the state variables and lead the policy to systematically withhold certain actions from an individual, unfairness will result.For example, Hispanics may under-report their pain levels due to cultural factors, misleading a fairness-unaware RL agent to assign less therapist time to these individuals (Piette et al., 2023).Deployment of RL algorithms without careful fairness considerations can raise concerns and erode public trust in high-stakes settings.