AI-Driven Auto-Architecting of Scalable Cloud Pipelines Using Workflow Analytics and Meta-Learning

With the rapid expansion of artificial intelligence (AI) across industries, there is a growing demand for intelligent cloud systems capable of adapting to dynamic workloads and scaling autonomously. Traditional cloud pipeline architectures remain static and manually configured, resulting in performance bottlenecks, inefficient resource utilization, and poor scalability. To overcome these limitations, this study proposes an AI-driven cloud pipeline architecture that leverages historical workflow analytics for the autonomous generation and optimization of cloud pipeline architectures. At its core, the proposed AI-driven cloud pipeline architecture introduces adaptive modular intelligence (AMI), which modularizes tasks based on historical patterns through a dynamic Infrastructure-as-Code (IaC) configuration strategy. It further integrates Flexible Batch Processing Scheduling (FBPS) to intelligently switch between batch and stream processing modes based on workload urgency and system load, enabling real-time scheduling flexibility. To enhance optimization, the Evolutionary Particle Genetic Optimizer (EPGO) refines scheduling configurations through a multi-objective fitness function balancing throughput, latency, and resource efficiency. In parallel, meta-orchestration system (MOS) manages encrypted communication between modules adopting symmetric key encryption, ensuring secure data exchange and protection against unauthorized access or tampering. MOS also tracks configuration changes and dynamically adapts pipeline behaviors without manual intervention, providing a robust governance layer for secure execution, real-time monitoring, and version-controlled reproducibility across the entire cloud infrastructure. Experimental results confirm that the proposed framework surpasses existing methods such as Genetic Algorithm (GA), Particle Swarm Optimization (PSO), Hybrid Genetic Algorithm (HGA), and Hybrid Parallel Enhanced Genetic Algorithm (HPEGA), achieving a minimum MAPE of 4.1%, faster convergence, improved scalability, and higher overall performance in dynamic AI-driven cloud environments.