Self optimizing Data Lakes

Enterprise cloud migration often creates static data platforms that cannot self-correct after go-live. We present the Self-Optimizing Data Lake (SODL), a technical architecture specification for a cloud-native data platform in which a machine-learning control plane continuously governs storage layout, compute allocation, and query execution across an AWS-native lakehouse. This paper's contribution is the design itself: a formally specified, implementable architecture that makes continuous cloud data platform optimisation practical in enterprise deployments where it has historically been manual and reactive. Three design elements constitute the core contribution: (i) a multi-objective optimisation formulation that navigates the cost-latency Pareto frontier across heterogeneous AWS services under a formal objective function with learned preference weights; (ii) a hybrid LSTM–Markov workload predictor that coordinates demand forecasts across query, ETL, and storage dimensions simultaneously; and (iii) a confidence-gated autonomy scheduler that maps prediction uncertainty to human oversight requirements, providing a principled mechanism for enterprise governance of autonomous system actions. The architecture is implemented on Amazon SageMaker Unified Studio using SageMaker Pipelines, Feature Store, Model Monitor, and Clarify. The implementation status of each component is documented explicitly in Section 5.5: some components were prototyped and validated on TPC-DS simulation data; others are design specifications awaiting production deployment. Conservative production-range estimates, derived from component-level evidence in the prior literature, suggest 30–60% query latency reduction, 40–70% ETL runtime reduction, and 20–35% storage cost reduction under typical enterprise conditions. Simulation results over a 500 TB TPC-DS synthetic corpus establish upper bounds of 85%, 91%, and 40% respectively under favourable conditions. The gap between conservative estimates and simulation upper bounds characterises the range of production outcomes and is itself a design-level finding.