Leveraging large language models for agentic process analytics assistants: Assessing accuracy in process mining tasks

To leverage Large Language Models for enhanced data-driven decision-making in Process Mining, this study introduces a framework that establishes an extended process analytics question taxonomy, including case, activity, resource, temporal, discovery and variant, and what-if and predictive dimensions. The proposed framework evaluates state-of-the-art Large Language Models on their ability to generate executable Python-based visual analytics, bridging the gap between raw Process Mining insights and non-technical users. Performance is assessed for recent models, such as GPT-4.1 and Claude 3.7, using an LLM-as-Judge approach and a specific Visualisation Error Rate. The framework works both as a benchmarking instrument and as a conceptual and practical basis to guide the development of deployable agentic assistants within AI-Augmented Business Process Management Systems. Results indicate that GPT-4.1 achieves on a bounded 1-10 evaluation scale (higher is better), an LLM-as-Judge score of μ = 8.18, σ = 2.3 and a Visualisation Error Rate of 25%. A key finding reveals a significant performance disparity: while models have an increased performance with widely adopted libraries, they face substantial challenges with domain-specific tools like PM4Py. These insights provide the necessary evidence to guide model refinement to fully support automated visual insight generation in Process Mining contexts, including the ones backed by other tools and languages, such as bupaR with R, and ProM with Java.