Entwicklung eines Systems auf Basis von Retrieval-Augmented Generation zur Unterstützung bei der Erstellung von sicheren Webapplikationen

Retrieval-Augmented Generation (RAG) has emerged as a promising approach for grounding large language models in authoritative knowledge sources. This work investigates how different RAG configurations perform when applied to security-critical knowledge derived from OWASP documentation. Based on the Design Science Methodology, a modular and containerized RAG architecture was developed to enable context-sensitive access to OWASP Cheat Sheets. For empirical evaluation, a domain-specific ground-truth dataset was constructed based on security topics prioritized by domain experts, with reference passages extracted verbatim from OWASP sources. The system is evaluated using two complementary benchmarking approaches. First, automated metric-based evaluation is conducted using the RAGAS framework, measuring context recall, context precision, answer relevance, and faithfulness. Second, a claim-based implementation of the PoLL framework is applied, employing a panel of heterogeneous language models (Mistral-7B, Command-R, and Qwen-3) to assess factual support at the statement level. The results show that retrieval quality is the dominant factor influencing overall system performance. Modern embedding models (BAAI/bge-small-en and infloat/e5-base-v2) significantly outperform all-MiniLM-L6-v2 across all metrics, with bge-small-en achieving the most stable balance between context precision and factual correctness. Among the evaluated language models, Mistral-7B exhibits the most consistent faithfulness scores, while smaller models show pronounced limitations when processing large or noisy contexts. A statistical comparison using the McNemar test reveals no significant difference between RAGAS faithfulness and PoLL-based judgments, indicating that the adapted PoLL approach yields results comparable to established metric-based evaluation methods. Overall, this work provides a reproducible RAG architecture, a transparent OWASP-based evaluation dataset, and an empirically validated benchmarking setup. The findings highlight the importance of embedding selection, demonstrate the value of combining metric-based and panel-based evaluation, and offer practical guidance for deploying RAG systems in security-sensitive development contexts.