llama-roofline: a portable memory-bandwidth roofline for llama.cpp
Abstract
A command-line tool that answers one question about a local LLM setup: what is limiting token generation on this machine? llama-roofline diagnose needs no configuration. It finds the user's GGUF models, measures the machine's sustainable memory-read bandwidth with a portable microbenchmark, benchmarks decode and prefill through llama-bench across a thread sweep, fits the decode roofline tok/s = BW_eff / bytes_per_token, and prints a plain-English report card: whether decode is bandwidth-bound, the token-per-second ceiling for that model, the thread count to use, and what using every thread costs. llama-roofline advise answers a second question, which quantization format to run, from a shipped matrix of controlled measurements across an Arm Cortex-A76, an Intel Golden Cove P-core and an Intel Gracemont E-core, including Raspberry Pi PMIC energy and perplexity on the same artifacts. The rankings differ by core, so the tool gives no single global ranking, states its evidence basis per core, and on unmeasured silicon declines to guess and shows how to measure instead. llama-roofline inspect prints a GGUF file's per-tensor type map and the bytes decode actually reads per token, which a format label does not describe. Outputs a roofline figure, Markdown reports for sharing, and machine-readable JSON. The analysis, the advisor and the GGUF reader are pure standard library; numpy and matplotlib are needed only to measure the bandwidth ceiling and to draw the figure.
Publication details
- Journal / source
- Zenodo (CERN European Organization for Nuclear Research)
- Publication date
- 27 August 2026