Fitting the Void: Residual-Aware Geometric Packing for GenAI Workloads

Generative AI workloads demand heterogeneous resources (GPUs, CPU cores, and high-bandwidth memory) in highly variable combinations. Current cluster schedulers, including Kubernetes, largely treat placement as scalar capacity packing (e.g., average utilization), which can mask infeasible nodes-servers may appear partially utilized while having no free GPUs, leaving CPUs or memory stranded. We propose Residual-Aware Geometric Packing (RAGP), a scheduling approach that matches each workload to a node’s residual multi-resource capacity. RAGP represents workload demand and server residual capacity as multi-dimensional vectors and uses cosine similarity to quantify alignment between a workload’s resource profile and a node’s remaining capacity.This geometric matching preferentially places CPU-intensive workloads on CPU-rich nodes, memory-intensive workloads on memory-rich nodes, and complementary workloads together, reducing fragmentation and improving effective utilization. To address bursty inference behavior, RAGP incorporates a predictive buffer: it forecasts short-horizon resource demand (including memory growth with longer conversations) and reserves headroom proactively rather than relying on instantaneous measurements.