A Novel Vision-Based Approach to Test Sequence Generation for Mobile GUI Testing

MobileGraphical User Interface(GUI) testing is a critical component of quality assurance for mobile applications. As GUI designs grow increasingly complex, vision-based testing methods have become essential for improving the quality of test scripts and reports. However, current approaches face significant limitations. For instance, the process of cropping GUI widgets requires significant manual effort and consumes considerable time. Meanwhile, existing automated testing tools still fail to generate satisfactory test sequences. In this paper, we proposeVision-based Test Sequence Generation(VTSG), a novel perception-driven approach for mobile GUI testing. More specifically: (1) VTSG employs a light-weight GUI element detection model to crop widgets from GUI pages automatically; (2) Guided by human perception principles, it sequences widget screenshots through saturation and spatial layout analysis; (3) The system then integrates GUI widget interaction instructions to generate vision-based test scripts that accurately simulate human interaction patterns on mobile devices. We evaluate VTSG against four state-of-the-art (SOTA) GUI testing tools across five applications. Experimental results demonstrate that VTSG significantly outperforms existing methods, achieving 47.44% code coverage and 51.33% activity coverage, respectively, compared to the other approaches. Additionally, we conduct a series of supplementary experiments on two mainstream commercial applications (i.e.,ToutiaoandDouyin). The results further confirm that VTSG maintains higher activity coverage even on these real-world commercial apps.