Morpheus: Lightweight RTT prediction for performance-aware load balancing

Distributed applications increasingly demand low end-to-end latency, especially in edge environments where co-located workloads contend for limited resources. Traditional load-balancing strategies are typically reactive and rely on outdated or coarse-grained metrics, often leading to suboptimal routing decisions and increased tail latencies. This paper presents a methodology for building and evaluating round-trip time (RTT) predictors that enable proactive and performance-aware load balancing. We develop fast, lightweight, and accurate RTT predictors that are trained on time-series monitoring data collected from a Kubernetes-managed GPU cluster. By leveraging a reduced set of highly correlated monitoring metrics, our approach maintains low overhead while remaining adaptable to diverse co-location scenarios and heterogeneous hardware. The predictors achieve up to 95% accuracy while keeping the prediction delay within 10% of the application RTT. In addition, we analyze the main contributing factors to prediction delay and provide insights for deploying effective predictors in resource-constrained clusters. Simulation-based analysis illustrates how these predictors can guide load-balancing decisions to reduce application RTT and resource waste, paving the way for their integration into real-time cluster management frameworks.