UML-to-Java Model Transformation with Large Language Models

Replication Package This repository contains the replication package for the research work: Benchmarking Large Language Models for Zero-Shot Model Transformation in Model-Driven Engineering The study evaluates whether pretrained Large Language Models (LLMs) can be used as direct, black-box model-transformation engines for UML-to-Java transformation. In the initial experimental condition, an LLM receives a UML class model together with a fixed transformation instruction and is asked to generate the corresponding Java artefact without transformation-specific implementation, source-target examples, explicit transformation rules, retrieved context, or fine-tuning. The benchmark contains 116 UML class models: 100 models represented in XMI, and 16 models represented in Ecore. Although Ecore is a metamodelling formalism rather than a UML serialization format, the 16 Ecore models encode class-model structures corresponding to the UML concepts considered in this study. They are therefore treated as UML class models within the benchmark while their Ecore representation is distinguished from the XMI representation used for the other 100 models. The evaluation uses three pretrained LLMs: GPT-5.2 DeepSeek v3 Gemini 2.5 Each source model is first processed using the same fixed zero-shot prompt. If the resulting Java artefact is not classified as error-free, one additional error-informed corrective generation is performed. The repository contains the scripts, benchmark artefacts, oracle outputs, generated outputs, comparison results, and supporting automation used in this experimental pipeline. Research Context and Goal Model transformation is a central activity in Model-Driven Engineering (MDE). Conventional transformation development typically requires explicit mappings, transformation languages such as ATL or QVT, metamodel knowledge, and transformation-specific tooling. This study investigates a different usage scenario: whether a pretrained LLM can be used directly as a black-box transformation engine by supplying a formal source model and a transformation request, without first constructing a transformation-specific implementation. UML-to-Java is used as the experimental transformation scenario. The objective is not to evaluate general-purpose Java code generation, but to assess whether the LLM-generated target artefacts preserve the transformation-relevant structure represented in the source models. The study examines: the proportion of source models for which each LLM produces an error-free Java result during the initial zero-shot generation; whether an unsuccessful initial transformation can be corrected after the LLM is informed of the detected transformation errors; how transformation success varies with source-model structural complexity; and which structural discrepancies occur most frequently in unsuccessful outputs. The reported results therefore characterize the evaluated LLMs under the documented experimental protocol rather than their maximum achievable performance under prompt engineering, examples, retrieval, fine-tuning, agents, or other forms of transformation-specific adaptation. Experimental Design The repository implements the experimental pipeline used in the study. The main stages are: Dataset collection and screening Oracle generation Initial zero-shot LLM generation Oracle-based comparison Filtering of presentation-level differences Error-informed corrective generation for unsuccessful initial outputs Result aggregation and error analysis The 116 source models are processed independently for each of the three evaluated LLMs. Two oracle-generation routes are used for the two subsets of the benchmark: Modelio for the 100 UML models represented in XMI; and ATL + Python for the 16 UML class models represented in Ecore. These oracle routes are applied to disjoint subsets of the benchmark. Generation Protocol The reported evaluation contains at most two generation attempts for each source-model-LLM pair. Attempt 1: Initial Zero-Shot Generation Every source model is first submitted using the same fixed transformation prompt: The following is an XMI file based on a UML diagram.Please write corresponding Java code without adding any explanatory text.Do not implement getters/setters and do not add comments.Generate classes as if they were in separate files.Start and end the response with [[STARTEND]] as a Java comment. The source artefact is supplied together with this instruction. The same wording is used for all 116 models. Consequently, the prompt describes the source as an XMI file even for the 16 Ecore inputs. This wording was intentionally kept fixed to control prompt variation across the benchmark. The resulting prompt-representation mismatch for the Ecore subset is documented as a limitation of the experimental protocol. During Attempt 1, the LLM receives no source-target examples, explicit transformation rules, explicit metamodel descriptions, oracle excerpts, retrieved contextual information, previous generated outputs, detected transformation errors, compiler diagnostics, or external tool results. Only Attempt 1 constitutes the zero-shot condition evaluated in the study. Attempt 2: Error-Informed Corrective Generation If the output produced in Attempt 1 is not classified as error-free, one additional generation is performed using the same source model and the same LLM. The complete initial transformation instruction is repeated, and the transformation-relevant error information detected through comparison with the oracle is appended to it. The corrective prompt follows this form: The following is an XMI file based on a UML diagram.Please write corresponding Java code without adding any explanatory text.Do not implement getters/setters and do not add comments.Generate classes as if they were in separate files.Start and end the response with [[STARTEND]] as a Java comment. In the initial run, the error is: [detected error description].In this run, please correct the identified error and generate the corresponding Java code again. The source artefact is supplied together with this prompt. Relative to Attempt 1, the source model and transformation instruction remain unchanged. The additional information consists of the detected transformation error and the request to correct it. No source-target examples, additional transformation rules, explicit metamodel descriptions, oracle excerpts, retrieved context, compiler diagnostics, or external tool results are introduced. Attempt 2 is therefore error-informed correction, not an independent second zero-shot sample and not a conventional pass@2 evaluation. Stopping Rule The reported evaluation stops after Attempt 2. An earlier version of the experimental pipeline supported a third generation. Exploratory execution showed that this third generation produced no additional outputs classified as error-free. The released replication pipeline and the reported evaluation therefore use at most two attempts. Because this stopping decision was informed by the exploratory observations, it should not be interpreted as evidence that additional corrective generations can never improve LLM-based model transformation. Prerequisites The experimental pipeline was executed on Windows using the following software and services: Windows AutoHotkey v1.1 and v2.0 Beyond Compare Visual Studio Code Language Support for Java™ by Red Hat v1.28.1 Python Google Chrome Modelio 4.1 ChatGPT through its web interface DeepSeek through its web interface Gemini through its web interface The browser-based automation uses AutoHotkey. During automated execution, interaction with the browser should therefore be avoided because keyboard focus, mouse interaction, or changes to the browser state can interfere with the automation. The evaluated LLMs were accessed through hosted web interfaces. Provider-side inference parameters such as temperature, top-p, random seed, and exact backend configuration were not exposed through these interfaces. Consequently, the repository supports replication of the documented experimental procedure and auditing of the reported artefacts, but exact regeneration of identical textual LLM outputs cannot be guaranteed. Installation and Configuration Java Formatting Configuration Enable automatic Java formatting in Visual Studio Code: Settings → Command Palette → Preferences: Configure Language Specific Settings → Java → Text Editor → Formatting → Format On Save Local Path Configuration Local executable paths must be configured before running the automation. For the Visual Studio Code executable path, configure: formatJava.ahk — lines 4 and 11 formatJava_SecondIteration.ahk — line 3 For the Python executable path, configure: firstIteration.bat secondIteration.bat specificExecution.bat Relevant configuration locations are also documented in the corresponding files. Beyond Compare Place a shortcut to Beyond Compare in the repository root directory so that the comparison scripts can invoke it using the expected path. Execution Demo Run To verify the local configuration, execute: specificExecution.bat and specify: demo The resulting artefacts are stored under: ./allXMI/works/demo/ The demo directory contains the source artefact, oracle output, LLM-generated output, and comparison information needed to inspect the workflow. Two-Attempt Workflow Attempt 1 Execute: firstIteration.bat This stage: supplies each source model to the selected LLM using the fixed zero-shot prompt; extracts the generated Java artefact; processes the generated files for comparison; compares the generated result against the corresponding oracle; and records whether transformation-relevant discrepancies remain. Outputs without retained transformation-relevant discrepancies are classified as error-free under the adopted evaluation criterion. Attempt 2 Execute: seco