Evaluating the Effectiveness of AI-Generated Test Cases for Industrial Applications

This paper examines whether three AI tools, GitHub Copilot, OpenAI ChatGPT and Google Gemini, can support commissioning engineers in the testing processes at the end of an industrial application’s installation. In the study, the above mentioned AI tools were used to generate test cases for a PLC control program that operates a 3D Delta Picker. The effectiveness of the test cases was evaluated both quantitatively and qualitatively. In the 3D Delta Picker case study, GitHub Copilot produced 10 standalone test cases, ChatGPT 5 suites with 12 test cases and Gemini 4 suites with 13 test cases, detecting 2, 4 and 2 of the injected mutants respectively and yielding at most 2 meaningless test cases per tool. Additionally, the AI tools were evaluated using a point-based scoring system. Mutation analysis was applied to measure the effectiveness of the test cases. Based on the results, it can be concluded that, with engineering validation, AI can provide effective assistance in testing an industrial application.