Balancing Accuracy and Cost: Trade‐Offs in Large Language Model Quantization for the Systems Engineering Domain

ABSTRACT As Large Language Models (LLMs) are increasingly deployed within the systems engineering domain, optimizing these models to balance performance accuracy and cost for given computational resources becomes essential. One process for finding the right balance is quantization, a process that involves converting model parameters from higher precision formats to lower precision formats. This empirical study investigates the impact of quantization at varying levels, including 4, 8, and 16‐bit floating‐point on LLM performance and cost within systems engineering problem contexts. The benchmark for systems engineering tasks, SysEngBench, is used to evaluate LLM performance. It is tailored to systems engineering domain knowledge and is mapped to the INCOSE Systems Engineering Handbook and IEEE 15288 System Life Cycle Processes, covering concepts across requirements, architecture, model‐based systems engineering, technical management processes, and specialty engineering activities. By examining accuracy loss across quantization levels, insights can be gleaned into scenarios where lower bit quantization provides resource efficiency gains with minimal impact on accuracy, as well as scenarios where higher bit quantization levels are paramount for maintaining high fidelity responses. Analysis on quantization levels versus accuracy and model size, together with inferencing computing costs, indicates distinct trade‐offs that can guide practitioners. These cost efficiency trade‐offs help in selecting appropriate quantization levels to balance desired performance, cost, and resource requirements, which is particularly useful for model deployment in computation‐constrained environments.