Large Language Models for Inter-Model Consistency Assessment

Cyber-physical systems are developed using heterogeneous modeling frameworks that address different engineering concerns and abstraction levels. In practice, models from mechanical, electrical, and software engineering coexist across development phases, creating the need to assess inter-model consistency despite differences in terminology, structure, and execution semantics. However, assessing consistency across independently designed modeling frameworks remains challenging due to thelack of explicit mappings and the multi-faceted nature of consistency. In this paper, we primarily investigate the semantic reasoning capabilities of Large Language Models (LLMs) to support inter-model consistency assessment, covering both a cold-start scenario without prior formal integration and a rule-grounded scenario derived from documented metamodel correspondences. We construct two manually curated gold-standard datasets: 90 automotive element mappings across four metamodels,including AUTomotive Open System ARchitecture (AUTOSAR), Robot Operating System 2 (ROS 2), Computer-Aided Design (CAD), and Simulink, and 32 smart-grid element mappings across three IEC metamodels, each annotated along three ordinal consistency dimensions: syntactic, semantic, and behavioral. Six large language models from OpenAI and Google Gemini are evaluated under zero-shot, one-shot, and few-shot prompting strategies. In addition, we evaluate two retrieval-enabled agenticcoding systems, OpenAI Codex and Anthropic Claude Code, to investigate whether autonomous retrieval and iterative reasoning improve inter-model consistency assessment. Model outputs are analyzed using ordinal-aware metrics, including Quadratic Weighted Kappa and Mean Absolute Error. The results indicate that the selected LLMs show systematic improvements over traditional automated heuristics (e.g., string-based, rule-based, and embedding-based baselines), particularly in semanticand behavioral consistency assessment. However, overall ordinal agreement remains weak to moderate across models and prompting strategies. Under the evaluated configurations and datasets, vendor-level aggregation reveals systematic differences, with Gemini models consistently achieving higher semantic and behavioral agreement, OpenAI models being slightly stronger syntactically, and the agentic Claude Code attaining the strongest overall agreement. Prompt augmentation returns incremental improvements but does not fundamentally alter performance trends. Overall, these findings suggest that current LLMs provide promising yet limited support for inter-model consistency assessment. While suitable as decision-support tools, significant advances are required before such models can be reliably integrated into fully automated model-driven engineering workflows. Reproducibility All experiments were conducted under controlled conditions: Fixed LLM model versions Standardized prompt templates Consistent inference parameters Structured JSON output enforcement Identical evaluation pipeline across models Vendor-level aggregation is computed as the arithmetic mean across all configurations, since each configuration is evaluated on the same dataset (50 instances). Detailed step-by-step reproduction instructions are provided in the included README file. Research Context This dataset enables controlled evaluation of LLM-based reasoning for multi-dimensional inter-model consistency in heterogeneous CPS modeling environments. The work contributes empirical evidence regarding: The comparative performance of LLMs versus traditional heuristic baselines The effect of prompting strategies on architectural reasoning Vendor-level differences in semantic and behavioral consistency assessment Limitations of LLMs in safety-critical modeling contexts To our knowledge, this represents one of the first systematic empirical evaluations of LLM-based reasoning for inter-model architectural consistency in heterogeneous CPS frameworks.