Evaluating the Effectiveness of AI-Mediated Emotional Support: A Comparative Analysis of Large Language Models for Loneliness and Stress Alleviation

Large Language Models (LLMs) are increasingly deployed for emotional support applications, yet systematic evaluation across different models remains limited. This study compares five LLMs (GPT-4 [1], Claude 3.5 Sonnet [2], Gemini [3], Llama 3-8B [4], Mistral [5]) on standardized emotional support scenarios using both automated metrics and human evaluation (N=30 responses, four criteria: empathy, helpfulness, appropriateness, user satisfaction). Automated evaluation revealed moderate performance (2.38- 2.79/5), with appropriateness highest (3.93-4.41) but empathy (2.01-2.61) and helpfulness (1.07-1.57) lower. Human evaluation produced substantially higher scores (2.75-4.13/5), with Claude 3.5 Sonnet [2] achieving the highest performance (4.13), followed by Mistral [5] (3.75) and GPT-4 [1] (3.46). The 1.0- 1.5 point discrepancy suggests that automated heuristics may underestimate the model’s capabilities. Validation strategies were nearly universal (96.7% automated, 100% human) yet did not ensure high empathy. Generic responses constituted the primary failure mode (13.3% automated, 6.7% human). Responses that combined validation with advice elicited higher empathy (2.77) than validation alone (2.36). While LLMs demonstrate baseline competence, improvements are needed in empathetic engagement and practical helpfulness. The disconnect between validation language and empathy scores indicates that supportive phrases alone do not guarantee perceived empathy. Human evaluation reveals that some models, particularly Claude 3.5 Sonnet [2], achieve better human-perceived quality than automated metrics indicate. This framework provides a standardized approach for model comparison, highlighting the need for specialized training and human oversight in mental health applications.