Evaluating Embedding Strategies for Retrieval-Augmented Generation on Turkish Customer Service Conversations

Retrieval-Augmented Generation (RAG) systems rely heavily on embedding models for effective document retrieval. While much progress has been made in English-language embeddings, less is known about their performance on morphologically rich, low-resource languages such as Turkish. This paper investigates the effectiveness of different embedding strategies for RAG on a domain-specific dataset of 1,247 Turkish customer-representative conversations (2,891 chunks) in the retail industry. We evaluate sixteen embedding configurations derived from four embedding models (OpenAI text-embedding-3-small, Qwen3-Embedding-0.6B, BERTurk, and BM25) across four data representations: (i) original Turkish text, (ii) English translation, (iii) Turkish keywords, and (iv) English keywords. A local Qdrant vector database with cosine similarity was used, and retrieval performance was evaluated on 500 hand-crafted customer queries using GPT-4 as an automated judge (92% human agreement). Results show that English-translated text embeddings yield the best performance across all models, with OpenAI achieving the highest accuracy (0.98), followed by Qwen (0.97) and BERTurk (0.95). All pairwise differences between top models were statistically significant (McNemar’s test, p < 0.05). BM25 provides a strong classical baseline (0.92 on English). These findings highlight the trade-offs between accuracy and latency in multilingual RAG systems.