Automatically Generating BPMN 2.0 Process Models from Natural Language Process Descriptions: Challenges, Framework, Quality Assessment

Abstract The increasing importance of process models highlights the demand for tools that enable an efficient and accurate transformation of natural language process descriptions into BPMN 2.0 process models. Despite significant advancements in natural language processing and large language models, automatically generating process models that are syntactically correct, semantically accurate, and pragmatically useful remains a challenge, particularly for individuals without deeper expertise in process modeling. To tackle this challenge, the authors have been developing BPMNGen, an LLM-based conversational framework, which leverages advanced natural language processing and large language models to automate the generation of BPMN 2.0 process models from natural language process descriptions. To evaluate the quality of the models generated with BPMNGen, two studies were conducted: one study focused on the semantic quality of the generated process models in relation to given process descriptions and another one assessed process model comprehensibility, including the cognitive load of the study participants when reading the LLM-generated models, their level of acceptability, and their performance. The results show that LLMs already perform well for simpler and moderately complex processes, while struggling with higher complexity – highlighting both current limitations and opportunities for further improving automated BPMN 2.0 model generation.