Will the EU AI Act help to mitigate dataset bias in medical AI?

The aim of this article is to provide an overview and analyze the implications of the provisions on dataset quality and bias in the AI Act (AIA). The AIA requires providers of AI systems to take measures to identify, prevent, and mitigate biases as part of the data governance practices. The AIA also explicitly prescribes certain characteristics required of training, validation, and testing datasets. These include notions widely considered as best practice such as representativeness as well as consideration of characteristics particular to the "geographical, contextual, behavioural or functional setting" which might expand the scope of considerations already common among AI developers. The AIA also aims to address the legal limitations on access to sensitive data by introducing the so called "debiasing exception," which under certain conditions permits the processing of sensitive data for debiasing purposes. To ensure enforcement of the data governance provisions, the AIA grants notified bodies and enforcement authorities access to training, validation, and testing datasets; however, further efforts may be needed to reconcile data protection concerns with these enforcement powers. The AIA's requirements will likely help mitigate bias in medical AI systems. Associated soft law instruments should contribute to the effective implementation of these requirements.