Prompt Recovery From Source Code: A Comparative Study of Large Language Models
Summary
No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.
Summarise with VIRAT Free to try with your Science Catalogs account.Abstract
This paper presents a rigorous empirical study on prompt recovery, the task of inferring the original natural language instruction from a given source code snippet. We evaluate four state-of-the-art Large Language Models (LLMs): OpenAI’s GPT model, Google’s Gemini, Meta’s Llama3, and DeepSeek’s DeepSeek-R1 on their ability to recover prompts from Python code. The analysis also investigates the impact of code degradation (via variable renaming and comment removal), which serves as a proxy for poorly documented legacy systems. Additionally a preliminary study of generative sampling temperature effects was conducted using OpenAI’s GPT model. We evaluate performance using a comprehensive suite of metrics, including BLEU, ROUGE, BERTScore, and Levenshtein Distance. Our findings reveal that while code cleaning degrades lexical similarity scores (e.g., average BLEU scores for OpenAI are 20% lower), it consistently maintains semantic similarity (e.g., average BERT-F1 remains above 0.90). This suggests that in the absence of lexical cues, models are forced to rely on deeper algorithmic reasoning and structural understanding of source code. Comparative analysis shows cloud-hosted models exhibit superior semantic fidelity, while DeepSeek-R1 demonstrates competitive character-level accuracy. These results provide critical insights into the code comprehension capabilities of modern LLMs and have significant implications for their application in reverse engineering, legacy system documentation, and the development of more robust evaluation benchmarks for code intelligence.
Fields of Study & Subjects
Software Engineering Research
Topic Modeling
Natural Language Processing Techniques
Publication details
- Journal / source
- IEEE Access
- Volume
- 14
- Pages
- 49714–49728
- Publication date
- 1 January 2026
- Language
- EN