The Role of Cognitive Abilities in Requirements Inspection: Comparing UML and Textual Representations - Replication Package

The present folder contains the supplementary material for the paper “The Role of Cognitive Abilities in Requirements Inspection: Comparing UML and Textual Representations” by Giovanna Broccia, Sira Vegas, and Alessio Ferrari. Context. The representation of requirements plays a critical role in the accuracy of requirements inspection. While visual representations, such as UML diagrams, are widely used alongside text-based requirements their effectiveness is still debated. Cognitive abilities, such as working memory and mental rotation, may also influence inspection accuracy, particularly depending on how the requirements are represented. Objective. This study aims to evaluate whether the use of UML sequence diagrams alongside text-based requirements improves the accuracy of requirements inspection compared to text-based requirements and to explore whether cognitive abilities are associated with differences in performance across the two types of representations (UML vs. text). Methods. We conducted a crossover experiment with 38 participants to assess the accuracy in finding issues under two treatments (text and text with UML support) using 2 tasks. Linear mixed-effects and generalized linear models were used to analyse treatment, period, sequence, and cognitive ability effects. Results. The results indicate a significant three-way interaction between representation type, working memory capacity, and mental rotation ability. This finding suggests that the effectiveness of UML is not uniform across individuals: participants with high scores in both cognitive abilities experienced reduced performance when using UML for defect detection. Conversely, the same cognitive profile was associated with improved justification accuracy under UML, indicating that high cognitive abilities may support deeper reasoning processes despite increasing task complexity. Conclusion. The findings suggest that while UML diagrams may not universally improve requirements inspection accuracy, certain cognitive profiles can benefit from them to improve inspection performance. These results highlight the importance of considering individual cognitive profiles when designing and assigning requirements inspection tasks. The folder contains the following sub-folders: 1. EXPERIMENTAL MATERIAL. The folder contains all the materials used in the experiment. Particularly, the folders containing all the materials provided to the participants in both groups on both days (i.e., Day1_Group1_Arkanoid, Day1_Group2_Arkanoid, Day2_Group1_Snake, Day2_Group2_Snake); the folder containing the post test questionnaires for both groups on both days (i.e., Post-test questionnaires); the folder containing the code to run both cognitive applications (i.e., COGNITIVE APPS). 2. GROUND TRUTH. The folder containing the xlsx files with the ground truth for Arkanoid and Snake requirements used in the experiment. 3. RAW RESULTS. The folder containing the test raw results (in the folder Test Raw results), the raw answers for the post-test questionnaires (in the folder Post-test questionnaire Raw), and the cognitive results (in the folder Cognitive Results). 4. DATA MAIN ANALYSIS. The folder containing the rmd file to run the analysis, the csv file containing the data to run the analysis, and the pdf knitted file. 5. POST-HOC ANALYSIS. The folder containing the rmd file to run the analysis, the two csv files containing the data to run the analysis, and the pdf knitted file. INSTRUCTION TO RUN THE ANALYSIS WITH R: 1. Download and install R from here: https://cran.rstudio.com/ 2. Download and install RStudio Desktop (Open Source Edition) from here: https://posit.co/download/rstudio-desktop/ --- ATTENTION: Always install the required additional packages if prompted --- 3. Open the rmd files with RStudio ("File > Open File"). The tool will display the analysis file in a window. 4. Run the analysis by clicking "Run > Run All" (top right row of the window). 5. Visualise the results (can be visualised in two ways): I. In a textual form: the tool will show the textual results in a separate window. II. In a separate HTML, PDF, or Word file with plots: click the "Knit" button in the analysis window. The file will be saved in the .rmd file directory.