Forecasting and Overcommit Pipelines for Cloud Storage: A Model for Persistent Disk Capacity Management

One of the major challenges for the administration of persistent disks, particularly in terms of disk size allocation to workloads is inefficiency. Thus, this paper reviews an implemented production-grade pipeline in forecasting storage demand and safe overcommitting of capacity. It replaces static, cluster-level hard-coded overcommit percentage with a dynamic, data-driven framework based on historical usage patterns and storage consumption to achieve a 35% improvement in correct resource allocation. This paper describes how to develop an operational model that takes historical usage data and current operational data and processes it through a statistical forecasting model to automate the decision-making process around ‘overcommitting’ resources. The model has been evaluated over multiple years on various regulated workloads at a large private cloud provider and has produced consistent improvements in allocation accuracy and operational safety. Results found that predictive resource forecasting and automated infrastructure control improved not only resilience in operation but also the use of resources within mission-critical cloud environments.