SAIL: Redesigning Collaborative Language Inference with a Single Server-to-Mobile Handoff

While on-device small language models (SLMs) enable responsive inference, they often lack the accuracy required for complex reasoning tasks; conversely, server-based large language models (LLMs) deliver high accuracy but incur prohibitive latency for real-time mobile applications. Collaborative inference approaches, such as split computing and speculative decoding, promise to bridge this gap, yet both require repeated network synchronization during generation, accumulating delays that negate their computational benefits. We present SAIL, a collaborative inference framework that achieves both high accuracy and low latency through a strategy we term Prefix Handoff Inference (PHI). In PHI, a server LLM generates the difficult early tokens and hands off completion to the mobile SLM, consolidating communication into a single handoff while providing sufficient context for the SLM to maintain accuracy. SAIL operationalizes PHI through three core modules: the handoff decision module determining the optimal prefix length tailored to each query's difficulty, the branch prediction module speculatively generating candidate continuations on-device while awaiting the server prefix, and the adaptive control module coordinating hand-off and branch prediction under fluctuating network and server conditions to meet service level objectives. Together, these modules enable SAIL to deliver up to 76% accuracy improvement over mobile SLMs with 98.9% server LLM accuracy retention, while strictly meeting latency constraints across translation, math, and expert level QA.