CoLoRA: A Collaborative Scheduling Framework for Multi-Tenant LoRA LLM Inference

Large Language Models (LLM) incur substantial resource costs during inference, driving widespread interest in Parameter-Efficient Fine-Tuning (PEFT) techniques. Among these, LoRA dramatically reduces overhead by updating only a few low-rank adapters. However, Multi-tenant LoRA LLM inference faces challenges from heterogeneous requests and latency-throughput trade-offs. Moreover, inefficient adapter reuse, poor cache management, and non-adaptive batching strategies severely restrict inference efficiency, service quality, resource utilization, and fairness. To address these challenges, we propose CoLoRA—a collaborative scheduling framework for multi-tenant LoRA LLM inference, comprising four core modules: (1) Adaptive Priority Scheduling (APS), which dynamically integrates queue waiting time, adapter residency status, and SLA urgency to compute task priorities; (2) Adapter-Aware Scheduling (AAS), which enhances cache management by prioritizing SLA-critical, frequently used, and fairly shared adapters, thus reducing cold-start latency and fragmentation; (3) Load-Aware Batch Scheduling (LBS), which combines real-time GPU utilization and queue depth to adaptively form batches and coalesce tasks targeting the same adapter, thereby improving parallelism while controlling latency; and (4) Unified Scheduler (US), which periodically gathers system metadata to orchestrate the submodules collaboratively and employs a feedback loop to optimize global strategies online. Evaluation on realistic multi-tenant workloads and popular open-source LLM shows that CoLoRA, compared to conventional baselines, increases overall system throughput by 56.5%, reduces P95 latency of online requests by 34%, and significantly enhances GPU utilization and tenant-level fairness, demonstrating its promise for large-scale inference services.