On-Device-First Hybrid LLM Inference on AI-PCs: Closing the Enterprise GenAI Divide

View at publisher Summarise with VIRAT
Authors: Sultana Begum, Kris Fleming, Todd Lewellen
Cited by: 0
References: 33

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

AI-PCs and modern PCs equipped with capable CPUs, GPUs, and NPUs now make on-device inference for small language models (SLMs) practical across many enterprise workloads. This enables assistants that are low-latency, privacy-preserving, and resilient to network conditions, while frontier cloud models still offer higher accuracy, richer tool use, and stronger multimodality than is practical to run on every endpoint. This paper surveys and organizes the design space for an on-device-first hybrid architecture in which an SLM on the AI-PC handles most requests by default, and escalation to cloud models is reserved for difficult, tool-heavy, or long-context tasks. We synthesize advances in compact local on-device optimized model design including architectures that mix self-attention with state-space models (SSMs), parameter-efficient adaptation, multimodal and structure-aware retrieval-augmented generation (RAG), and long-horizon memory, and connect these to hardware and serving optimizations such as quantization, mixed precision, and KV-cache management. Building on this, we propose routing and reporting patterns that make latency, energy, cost, and privacy explicit design parameters, and outline a total-cost-of-ownership view that clarifies when AI-PC–centric deployments outperform all-cloud strategies. We argue that such edge-first hybrid designs offer a practical path to narrowing the "GenAI Divide," in which impressive pilots fail to translate into durable, production-grade business impact.

Fields of Study & Subjects

Topic Modeling IoT and Edge/Fog Computing Advanced Neural Network Applications

Publication details

Journal / source
International Conference on Consumer Electronics
Pages
1–6
Publication date
3 February 2026
Language
EN