Empirical Evaluation of AI-Assisted Risk Reasoning for ICPS Software Security

Operational Technology (OT) systems form the control and actuation layer of Industrial Cyber-Physical Systems (ICPS), linking computational intelligence with physical processes. Typical OT components (e.g., PLCs) with embedded dedicated control software operate in safety-critical environments where software reliability and cybersecurity are essential for overall system trustworthiness. However, existing OT security resources emphasize taxonomies and threat catalogs rather than reproducible benchmarks for empirical risk reasoning, particularly in evaluating Artificial Intelligence (AI)-assisted methods for cybersecurity risk assessment. In this context, this study introduces a reproducible, standards-aligned benchmark designed to evaluate AI-assisted risk reasoning in OT software within ICPS environments. This paper presents two main contributions toward advancing trustworthy and reliable ICPS. The first is the design ofEmbSoftOTBench, a standards-aligned benchmark dataset for ICPS software risk reasoning that operationalizes IEC 62443-3-2 through machine-readable scenarios derived from MITRE EMB3D Embedded Systems Threat Matrix (EMB3D). The benchmark provides paired threat and benign scenarios with quantitative factors and expert rationales, enabling structured and reproducible evaluation of AI-assisted security reasoning within ICPS engineering workflows. The second contribution is an experimental evaluation of state-of-the-art Large Language Models (LLMs) employing the benchmark dataset. Results reveal systematic trade-offs between sensitivity, specificity, and calibration, establishing first baselines for AI-empowered ICPS software risk reasoning. The dataset is publicly released under the CC BY 4.0 license to support reproducible research, engineering practice, and education in safety-critical ICPS environments.