Fairness Evaluation in LLM-Based Educational Recommender Systems: A Framework for Equitable Learning

The integration of large language models (LLMs) into recommender systems has raised significant concerns about fairness, particularly in educational contexts where equitable access to learning resources is crucial. To address this, we introduce FairEval, a novel evaluation framework designed to systematically assess fairness in LLM-based recommendations by incorporating both sensitive demographic attributes (eight categories spanning 31 values) and personality profiles. FairEval introduces a new metric, @25, to quantify recommendation consistency under identity and personality variations. We demonstrate the applicability of FairEval by evaluating two state-of-the-art LLMs – ChatGPT 4.0 and Gemini 1.5 Flash – on movie and music recommendation tasks, using a structured prompt set of over 1,000 neutral and sensitive user personas. Our findings reveal significant disparities. The evaluation of ChatGPT-4.0 and Gemini 1.5 Flash highlights the importance of considering personality-aware fairness in LLMs. For instance, the results for ChatGPT-4.0 show that the similarity scores for the continent attribute are 0.7840 (max) and 0.7549 (min), with sensitive-to-neutral similarity range (SNSR) at 0.1900 and sensitive-to-neutral similarity variance (SNSV) at 0.0770. Similarly, Gemini 1.5 Flash has a maximum similarity of 0.7566 and a minimum similarity of 0.7404, with SNSR at 0.1363 and SNSV at 0.0507. Despite these promising results, fairness gaps persist, with SNSR disparities for religion and race continuing to highlight bias. By integrating personality-aware fairness evaluation into the LLM-based recommender pipeline, FairEval offers a robust benchmark for developing inclusive and bias-aware LLM-based recommender systems, with particular relevance to educational applications.