LLM-based text-to-JQL Generation: an automated test using execution and structural indicators
Summary
No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.
Summarise with VIRAT Free to try with your Science Catalogs account.Abstract
Evaluating natural language to query language (NL-to-QL) systems cannot rely on string matching alone, especially when queries involve dynamic data or complex query languages. We present APOLLO, an automated test framework that executes generated queries by Large Language Models (LLMs) against a live Jira instance, enabling execution-aware evaluation and detailed error diagnosis across syntax, unsupported features, and value inaccuracies. APOLLO introduces new metrics—field accuracy, value accuracy, and inference accuracy—that capture correctness beyond similarity. Using APOLLO, we evaluated two LLM-based conversion architectures: a retrieval-augmented generator and a specialized API-supported generation model that demonstrated both limitations and alternatives to address the drawbacks of generating dynamic non-pre-trained data. Future work will extend the framework to additional query languages, explore automated remediation strategies, and investigate richer semantic metrics for deeper diagnostic insight.
Fields of Study & Subjects
Software Engineering Research
Natural Language Processing Techniques
Topic Modeling
Publication details
- Journal / source
- International Conference on Consumer Electronics
- Pages
- 1–6
- Publication date
- 3 February 2026
- Language
- EN