LLM-based text-to-JQL Generation: an automated test using execution and structural indicators

View at publisher Summarise with VIRAT
Authors: Rayfran Rocha Lima, Irineu Evangelista Cruz de Brito, Jardel da Cunha Nascimento, Luiz Alberto Queiroz Cordovil Júnior
Cited by: 0
References: 28

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

Evaluating natural language to query language (NL-to-QL) systems cannot rely on string matching alone, especially when queries involve dynamic data or complex query languages. We present APOLLO, an automated test framework that executes generated queries by Large Language Models (LLMs) against a live Jira instance, enabling execution-aware evaluation and detailed error diagnosis across syntax, unsupported features, and value inaccuracies. APOLLO introduces new metrics—field accuracy, value accuracy, and inference accuracy—that capture correctness beyond similarity. Using APOLLO, we evaluated two LLM-based conversion architectures: a retrieval-augmented generator and a specialized API-supported generation model that demonstrated both limitations and alternatives to address the drawbacks of generating dynamic non-pre-trained data. Future work will extend the framework to additional query languages, explore automated remediation strategies, and investigate richer semantic metrics for deeper diagnostic insight.

Fields of Study & Subjects

Software Engineering Research Natural Language Processing Techniques Topic Modeling

Publication details

Journal / source
International Conference on Consumer Electronics
Pages
1–6
Publication date
3 February 2026
Language
EN