De-Quantization Penalties for Interactive LLM Inference on Prosumer GPUs

Post-training 4-bit quantization is often treated as the default path for running large language models (LLMs) on prosumer GPUs: if decoding is memory-bandwidth bound, shrinking weights from FP16 to 4-bit should cut memory traffic and improve latency and energy efficiency. We revisit this assumption on an RTX 3090 (Ampere) that lacks native INT4 tensor support. For 1–8 billion parameter models (TinyLlama, Qwen-2.5, Mistral, Llama 3.1, DeepSeek-R1-8B), we compare native FP16 inference against AutoGPTQ 4-bit models and GGUF kernels inllama.cpp. On a standard Transformers+AutoGPTQ stack, interactive batch size$B=1$decoding is still1.3–2.2× slowerthan FP16 despite a 2.4× reduction in VRAM usage, and some INT4 configurations are up to2.4× less energy-efficient. An optimized GGUF backend improves 4-bit TinyLlama throughput by 1.65× over GPTQ, indicating that the de-quantization penalty is dominated by kernel design rather than hardware limits. We conclude that on prosumer GPUs without native INT4 tensor cores, 4-bit quantization is only attractive when paired with mature low-bit kernels; otherwise, FP16 remains the more robust choice for interactive workloads.