TokenPool-Scheduler: Token-Level GPU Pooling and Resource Slicing for Multi-Model Co-Location

In the context of GPU scarcity, multi-model and multi-tenant inference services for autoregressive large language models require finer-grained resource sharing to improve utilization and throughput. Existing approaches rely on coarse-grained spatial or temporal multiplexing, which often leads to suboptimal GPU utilization and increased tail latencies. We present TokenPool-Scheduler, a token-level GPU pooling and resource slicing framework that breaks LLM inference execution into fine-grained segments. By employing a token-level cost model and adaptive scheduling policies, TokenPool-Scheduler dynamically determines optimal slice sizes, preemption points, and resource quotas for heterogeneous models. Experimental results demonstrate that TokenPool-Scheduler achieves up to 33% higher GPU utilization and 63% improvement in goodput compared to static allocation schemes, while reducing P99 time-to-first-token (TTFT) latency by 43% under high load (250 req/s per GPU). The system reduces model switching overhead by 69% compared to existing spatial-temporal multiplexing approaches.