UQA-RAG: Enhancing Urdu Question-Answer Retrieval With Semantic Similarity and LLMs

Urdu Question Answering (QA) systems struggle with limited annotated resources and linguistic complexities. These are significant hurdles for traditional Large Language Models (LLMs) that rely on static, pre-trained knowledge. This paper introduces UQA-RAG, a new framework to enhance Urdu QA by using Retrieval-Augmented Generation (RAG) over the multi-domain UNQA dataset. Our core methodological contribution is a Hybrid Retrieval Strategy. This approach overcomes the individual weaknesses of pure semantic or lexical search by combining their strengths. It fuses results from OpenAI’s text-embedding-3-small vectors (Semantic) with BM25 (Lexical) using the Reciprocal Rank Fusion (RRF) algorithm. This method maximizes both context relevance and diversity. We conducted an extensive ablation study to evaluate UQA-RAG’s performance with different LLM backbones, including GPT-4o, Llama-3.2, and Deepseek-r1. The optimal setup, pairing Hybrid Retrieval with GPT-4o, achieved a BERTScore F1 of 93.15%. This was a 16.55 percentage point improvement over our fine-tuned mBERT baseline. The system also reached a high RAGAS Context Precision of 95.20%. These results confirm that our hybrid approach is essential for generating accurate, contextually grounded responses, making UQA-RAG an effective solution for advancing high-quality QA in Urdu.