Prefill Is the Bottleneck: An Empirical Study of CPU-Only On-Device LLM Latency and One Lever That Exploits It

Sovereign, offline-first AI increasingly runs large language models (LLMs) on commodity CPUs with no GPU, yet latency intuition remains grounded in GPU serving, where autoregressive decode dominates. We instrument end-to-end inference of a quantized 7B model (Qwen2.5-7B, Q4) on a single CPU-only machine (AMD Ryzen 7 8845HS, 8 cores / 16 threads, 14 GiB) and sweep prompt length from 55 to 2,336 tokens (5 repetitions per point). Prefill — reading the prompt — grows from 38% of end-to-end latency at 55 tokens to 95% at 2,336 tokens, crossing 90% for prompts beyond ~1,300 tokens. Prefill time scales with prompt length while decode stays roughly flat (~11-12 tok/s), so end-to-end latency on CPU is governed by context length, not output length — inverting the GPU-era assumption. The effect is not model-specific (Llama-3.1-8B and Gemma2-9B also sit at 95-96% prefill at ~1,300 tokens) and reproduces on a second CPU vendor (Intel Comet Lake). We further find prefill speed is non-monotonic in quantization: the 8-bit build prefills fastest, inverting the "quantize-down-for-speed" heuristic on CPU. Finally, gating a second "review/verify" model pass cuts latency 2.4x with no measurable quality loss under an order-swapped LLM judge. We release all harnesses and raw data.