Memory-aware Hierarchical Scheduling of ML Tasks in Heterogeneous Clusters of Edge Computing Devices with GPUs

Machine Learning (ML) workloads represent a major share of edge computing tasks, where efficient scheduling is critical to exploit heterogeneous edge devices’ capabilities in terms of their GPUs. We find that conventional schedulers allocate resources opportunistically, focusing on availability rather than optimizing the overall execution time of ML jobs. This results in inefficient utilization of heterogeneous resources. An example is a heuristic-based strategy of First Come First Serve (FCFS), combined with Least Loaded First. In this approach, jobs are assigned in an FCFS manner and placed on the least loaded resourceful device. This allocation blocks subsequent larger tasks requiring high resources, thereby increasing the completion time of all tasks, known as the makespan [1]. To alleviate the problem, we propose MemBatch ILP, an Integer Linear Programming (ILP)-based scheduler that performs memory-aware batching to minimize the makespan of ML workloads while reducing the idling of GPUs. We employ a hierarchical scheduling technique – memory-aware batching at the cluster level and fine-grained task-splitting at the device level, enabling parallel execution and balanced utilization of heterogeneous resources. We compare our MemBatch ILP with a baseline ILP scheduler and FCFS-Least Loaded First on a testbed of heterogeneous Jetson edge devices. Our results show that MemBatch ILP reduces the makespan by up to 48% compared to FCFS-Least Loaded First and 61% compared to baseline ILP.