On-Device Small Language Model Inference Acceleration for Real-Time Robotics: A Survey

Small language models (SLMs) and on-device large language model (LLM) deployments are rapidly becoming key enablers of natural language interaction, high-level decision making, and semantic perception in robotics. Unlike datacenter-scale models, robotic deployments must satisfy tight latency, energy, and memory constraints while operating on embedded platforms such as NVIDIA Jetson, mobile systems-on-chip (SoCs), or custom controllers, often without reliable network connectivity. This paper surveys inference acceleration methods for on-device SLMs with a focus on real-time robotic use cases. We organize techniques into four layers: (i) model-level approaches such as low-bit quantization, distillation, architecture optimization, and sparsity; (ii) algorithmic methods including efficient attention, KV-cache compression, and speculative and multi-token decoding; (iii) system-level runtimes and scheduling techniques for embedded and edge platforms; and (iv) hardware trends spanning mobile neural processing units (NPUs), Jetson-class GPUs, FPGAs, and vision-language-action (VLA) accelerators. We highlight representative state-of-the-art SLM families (e.g., Llama 3.2, Phi-3, Gemma, Qwen 2.5, MobileLLM, MobileLLMPro) and robotics-oriented models such as Gemini Robotics ondevice and edge LLMs for ground robots. We also summarize design trade-offs through a comparison of inference frameworks (ExecuTorch, MLC-LLM, NanoLLM, llama.cpp, MediaPipe LLM, TensorRT-based stacks) and discuss open challenges for deploying language models as real-time components in robotic systems.