Human-in-the-Loop AI for Cloud Data Engineering: The Collaborative Intelligence Architecture (CIRA) for Regulated Industries

The technologically progressive nature of the Economy (including; healthcare delivery systems, drug discovery/development practices etc.) has resulted in greater interest amongst industry practitioners to modernise these sectors by utilising AI-enabled /digitally-enhanced technologies that will provide them with improved operational efficiency. However, current automation-oriented models do not adequately reflect the architectural features required for accountability, auditability, and human-machine interface (HMI) to support oversight by a person or trained professional of all automated systems within known regulatory environments. This paper introduces the Collaborative Intelligence Architecture (CIRA), which is a governance-focused and human-centric AI-based approach to developing and implementing data engineering workflows designed to provide expert validation, explainability, and override authority directly into the cloud data workflow. In contrast to existing models that retrofit the role of the human supervisor after the deployment, CIRA incorporates human judgement as a fundamental design principle in its architecture. The results obtained from applying this framework demonstrate a number of measurable benefits including 30-45% decrease in the amount of time required to manually process information, 25-40% decrease in the number of compliance-related errors, and 2-4 times shorter timeline for generating analytical reports, while also complying with applicable legal and regulatory requirements. This research provides a repeatable model for responsibly implementing AI within a regulated cloud computing environment, which supports broader efforts at the national level to enhance the safety of healthcare, ensure the integrity of science, and promote the resiliency of the financial system.