SOAR: Evaluating the reasoning capabilities of large language models on IT operational data

This work introduces System Operations and Reasoning Benchmark (SOAR), a novel benchmark designed to evaluate the reasoning capabilities of large language models (LLMs) in the context of IT operations and system administration tasks. The benchmark focuses on two complementary perspectives: the cognitive abilities required to infer or deduce information from operational data, and the formulation of the request without relying on complex prompting.The dataset comprises 315 questions across 82 content items extracted from real operational data, organized into domains such as system information, package management, services, networking, and containerization. Each question is cognitively categorized according to Bloom’s Taxonomy and the Depth of Knowledge (DOK) framework.Experimental results across multiple open and proprietary models reveal that response accuracy decreases as the cognitive complexity of the questions increases, while semantic quality (as measured by BERTScore) remains high. Verification-type questions expose limitations in handling non-existent or contradictory information. Moreover, most model outputs exceed the expected textual scope as reflected by low Exact Match scores. These results highlight the need for developing or adapting models specifically targeted at operational reasoning tasks.