Fragmentation-Aware and Efficiency-Oriented Scheduling for GPU Sharing Workloads

The rapid development of Artificial Intelligence (AI) has increased the demand for GPU resources, leading to a surge in deep learning tasks on heterogeneous GPUs. As a result, GPU resources in production clusters are often limited. It is essential to execute deep learning tasks efficiently within these resource and time constraints. Reducing task completion time (TCT) and improving GPU utilization are both critical. However, these goals often conflict, and existing solutions remain basic, making joint optimization challenging. We propose BFE, a method that integrates fragmentation awareness and efficiency orientation. BFE improves GPU utilization and reduces TCT by utilizing a cost model that combines average TCT and GPU fragmentation. We introduce a new fragmentation metric that considers both task size and GPU heterogeneity. For heterogeneous environments, we present a mathematical formulation that further enhances GPU efficiency and task scheduling. We also present a sliding window-based strategy to support the online scheduling scenario. We then design a particle swarm optimization-based scheduling algorithm to minimize the combined cost. Experiments on Alibaba's production traces demonstrate that the proposed BFE improves the GPU utilization by up to 51.73% and reduces the average TCT by up to 55.16%, outperforming existing methods.