From Accuracy to Insight: Explainability in Review Rating Prediction with Transformers

Mobile application (app) reviews provide valuable information that facilitates understanding of users’ needs, leading to better design of developed products. They have abundant data that can be utilized by different models to explain the prediction results to stakeholders. This will lead mobile app developers to trust and rely on the models that are used to develop their apps and satisfy the users’ needs. To leverage this information, outstanding improvements in complex learning algorithms have led to the development of transformer-based models that are used for natural language processing (NLP) and to exploit rating predictions. However, such models are complex and lack explainability, especially for Arabic reviews. Most studies have applied explainability models for transformer-based models to the English language and various other languages but not the Arabic language. This study presents a rating prediction explain-ability (RPE) framework that combines transformer-based and explainability models for review rating predictions from mobile government (m-government) apps. The transformer-based models predict the ratings for reviews written in English or Arabic. Then, local explainability models, such as SHapley Additive exPlanation (SHAP) and local interpretable model-agnostic explanations (LIME), explain and visualize the results. In RPE, not only high prediction accuracy was achieved for both English and Arabic reviews, but the resulted predictions were also justified with consistency between the different explainability models. The transformer-based model ELECTRA yielded the highest accuracy and F1 score of 96% for the rating prediction of English reviews, whereas the transformer-based model AraBERTv2 had 95%accuracy and F1 score for the rating prediction of Arabic reviews. The results of both explainability models provided equivalent explanations and emphasized the same words that affected the predicted ratings.