Calibration Under Extreme Imbalance: A Multi-Cluster Benchmark for Operational Queue Delay Prediction

When you submit a job to a shared GPU cluster, you often have no idea how long you'll wait. We investigated whether queue delays can be predicted using only information available at submission timebefore the job actually starts running. We tested three production traces: 7,386 GPU training jobs from Alibaba, 512 pods from our EKS cluster, and 20,000 tasks sampled from Google Borg. The answer depends on feature availability. With detailed resource specs and cluster state, logistic regression hit AUROC 0.97 with calibrated probabilities (ECE 0.041). Job attributes alone dropped AU-ROC to 0.87. Google's minimal Borg metadata gave near-random results (AUROC 0.53). Model choice didn't matter. We compared five classifiers-logistic regression through XGBoostand McNemar's test found no significant differences on any dataset. What limited accuracy was feature availability, not model choice. Isotonic calibration reliably reduced miscalibration by up to 48% without hurting discrimination. We describe the methodology in detail: how to split data to avoid leakage during calibration, how to compute unbiased calibration metrics, and how to properly compare classifiers under severe class imbalance. The approach should be useful for anyone building probability-based predictions in operational settings.