Evaluation of a Retrieval Augmented Generation approach for technical documentation in manufacturing

Recent advancements in Generative Artificial Intelligence (GenAI), driven by the development of Large Language Models (LLMs), have created opportunities for innovative solutions across multiple sectors, including manufacturing. However, deploying LLMs in industrial settings presents significant challenges due to their general-purpose nature, which often results in inaccurate or irrelevant responses when applied to domain-specific tasks. This paper explores the integration of Retrieval-Augmented Generation (RAG) to improve the performance of LLMs in industrial environments. The study investigates the effectiveness of five LLMs (GPT-4o, Mixtral-8x22B-Instruct-v0.1, Llama-3.3-70B-Instruct, DeepSeek-V3, Qwen2.5-72B-Instruct) in processing industrial technical documentation. The models’ performance was evaluated with and without RAG augmentation, using a dataset of 100 verified FAQs from an industrial machine manual. Evaluation criteria included expert evaluation, embedding-based evaluation, and comparative LLM assessment. Our findings show that RAG significantly improves the accuracy and relevance of LLM responses, achieving over 93% accuracy on general FAQs and more than 83% on specific, domain-dependent queries. The study demonstrates that RAG-based architectures provide a scalable, flexible solution to adapt LLMs to specialized manufacturing contexts. This research contributes to bridging the gap in understanding how to best implement LLMs in industrial applications and offers valuable insights for future research and practical implementations in Industry 5.0.