Transformer-Based Classification of Security-Related Mobile App Reviews Using Active Learning

Mobile app user reviews provide valuable insights into software functionality, quality, and security. However, existing approaches for identifying security-related reviews often rely on traditional feature extraction methods, limiting their ability to capture nuanced and implicitly expressed security concerns. This study presents an automated framework for classifying mobile app user reviews into security-related and non-security-related categories. The proposed framework leverages semantic representations and an iterative active learning process to improve annotation efficiency and model performance. A web-based graphical user interface (GUI) integrated with a large language model (LLM) is developed to provide interpretable explanations for model predictions. We collected over one million reviews and applied a three-stage annotation pipeline consisting of keyword-based filtering, expert validation, and iterative active learning using sentence-transformer embeddings enhanced with Low-Rank Adaptation (LoRA). Using the resulting labeled dataset, two transformer-based models (SBERT and Paraphrase) were fine-tuned for binary classification. Under 10-fold cross-validation, SBERT achieved an accuracy of 88.57% and an F1-score of 88.65%, while the Paraphrase model achieved 90.38% accuracy and 90.40% F1-score. On the held-out test set, SBERT obtained 71.43% accuracy and 70.98% F1-score, whereas the Paraphrase model achieved 91.07% for both accuracy and F1-score. These results demonstrate that transformer-based models leveraging sentence embeddings are effective for identifying security-related user reviews, with the Paraphrase model providing superior and more consistent generalization. The integrated GUI and LLM further enhance interpretability, offering a practical tool for supporting security assessment in mobile applications.