Automated Quantum Issue Labeling in Qiskit: Large Language Models and Fine-Tuned Transformers

# Automated Quantum Issue Labeling in Qiskit: Large Language Models and Fine-Tuned Transformers This repository contains the implementation code for the paper **"Automated Quantum Issue Labeling in Qiskit: Large Language Models and Fine-Tuned Transformers"** submitted to EMSE (Empirical Software Engineering, Springer). The code includes experiments comparing fine-tuned transformer models (DistilBERT, RoBERTa), Large Language Models (GPT-4o Mini, GPT-5 Mini, GPT-5 Nano), and RAG-enhanced LLM pipelines on quantum computing software issue classification. ## Table of Contents- [Study Overview](#study-overview)- [Repository Structure](#repository-structure)- [Installation](#installation)- [Configuration](#configuration)- [Usage](#usage)- [Input Data Format](#input-data-format)- [Output and Results](#output-and-results)- [Analysis Tools](#analysis-tools)- [GPU Acceleration](#gpu-acceleration-notes)- [Computational Requirements](#computational-requirements)- [Hyperparameters](#hyperparameters)- [Troubleshooting](#troubleshooting)- [Citation](#citation)- [License](#license) ## Study Overview This study compares the performance of multiple approaches for classifying GitHub issues in quantum computing repositories: **Fine-tuned Transformer Models:**- DistilBERT (F1=0.95)- RoBERTa (F1=0.94) **Large Language Models (direct prompting, 242-issue test set):**- GPT-5 Mini — zero-shot (F1=0.77) and few-shot (F1=0.82)- GPT-5 Nano — zero-shot (F1=0.59) and few-shot (F1=0.65)- GPT-4o Mini — zero-shot (F1=0.62) and few-shot (F1=0.64) **RAG-Enhanced LLM Pipelines (with threshold tuning, 721-issue test set):**- Agentic RAG + GPT-4o Mini: zero-shot F1=0.606, few-shot F1=0.682- Adaptive RAG + GPT-4o Mini: zero-shot F1=0.613, few-shot F1=0.744- Adaptive RAG + GPT-5 Mini: F1=0.836- Direct GPT-5 Mini few-shot (no RAG, same 721-issue test set): F1=0.843 We evaluate models on their ability to automatically classify Qiskit GitHub repository issues across 12 quantum module labels (e.g., `mod: circuit`, `mod: transpiler`). ## Repository Structure ```.├── Data/│ └── qiskit_repo_quantum_issues.json # 2,415 labeled Qiskit issues│├── Analysis/│ ├── predictions/│ ├── results/│ ├── bot_labeling_analysis.py│ ├── config.py│ ├── ml_baselines.py│ ├── quantum_term_analysis.py│ ├── statisticalanalysis.py│ └── analysis_requirements.txt│├── Fine_tuned_Experiments/│ ├── output/│ ├── distilbert_final.py│ ├── finetunedconfig.py│ ├── roberta_distilbert_requirements.txt│ └── roberta_final.py│├── Gpt_Experiments/│ ├── scripts/│ │ ├── gpt4o_mini_fewshot_gridsearch.py│ │ ├── gpt4o_mini_zeroshot_gridsearch.py│ │ ├── gpt_5_mini_fewshot_gridsearch.py│ │ ├── gpt_5_mini_zeroshot_gridsearch.py│ │ ├── gpt_5_nano_fewshot.py│ │ └── gpt_5_nano_zeroshot.py│ ├── config.py│ └── gpt_requirements.txt│├── RAG_Experiments/│ ├── code/│ │ ├── 01_agentic_rag_zeroshot.py # Agentic RAG, GPT-4o Mini, zero-shot│ │ ├── 02_agentic_rag_fewshot.py # Agentic RAG, GPT-4o Mini, few-shot│ │ ├── 03_adaptive_rag_zeroshot.py # Adaptive RAG, GPT-4o Mini, zero-shot│ │ ├── 04_adaptive_rag_fewshot.py # Adaptive RAG, GPT-4o Mini, few-shot│ │ ├── 05_adaptive_rag_gpt5mini.py # Adaptive RAG, GPT-5 Mini, few-shot│ │ ├── 06_threshold_tuning.py # Per-label threshold tuning utility│ │ ├── 07_gpt5mini_fewshot_direct.py # Direct GPT-5 Mini baseline (no RAG)│ │ └── config.py # RAG-specific configuration│ ├── predictions/ # Saved prediction JSON files│ ├── results/ # Threshold tuning result JSONs│ ├── rag_requirements.txt│ └── README.md│├── finetuned_distilbert_results/│ └── DistilBERT_Results/│├── finetuned_roberta_results/│ └── RoBERTa_Results/│├── gpt_4o_mini_results/│ ├── few_shot_results/│ ├── visuals_insights/│ └── zero_shot_results/│├── gpt_5_mini_results/│ ├── few_shot_results/│ └── zero_shot_results/│├── gpt_5_nano_results/│ ├── few_shot_results/│ └── zero_shot_results/│├── .gitignore├── LICENSE└── README.md``` ## Installation ### Requirements- Python 3.8+- CUDA-compatible GPU (recommended for fine-tuning experiments)- PyTorch- OpenAI API key (for GPT and RAG experiments) ### Setup 1. **Clone the repository:**```bashgit clone [repository URL]cd quantum-bug-labeling-main``` 2. **Create and activate a virtual environment:**```bash# For Windowspython -m venv venvvenv\Scripts\activate # For macOS and Linuxpython3 -m venv venvsource venv/bin/activate``` 3. **Install Fine-tuned experiments requirements:**```bashcd Fine_tuned_Experimentspip install -r roberta_distilbert_requirements.txt``` **Note**: The `roberta_distilbert_requirements.txt` includes:- `torch>=1.12.0` — PyTorch deep learning framework- `transformers>=4.30.0,=1.24.0`, `pandas>=2.0.0` — Data handling- `scikit-learn>=1.2.0` — Machine learning metrics- `matplotlib>=3.7.0`, `seaborn>=0.12.0` — Visualizations- `tqdm>=4.65.0` — Progress bars- `datasets>=2.12.0` — Dataset utilities 4. **Install PyTorch (for Fine-tuned experiments):** Option 1: CPU-only (simpler but slower for training)```bashpip install torch torchvision torchaudiocd ..``` Option 2: GPU with CUDA 11.8 (recommended)```bashpip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118cd ..``` 5. **Install GPT experiments requirements:**```bashcd Gpt_Experimentspip install -r gpt_requirements.txtcd ..``` 6. **Install RAG experiments requirements:**```bashcd RAG_Experimentspip install -r rag_requirements.txtcd ..``` **Note**: The `rag_requirements.txt` includes:- `openai>=1.13.3` — OpenAI API client (GPT-4o Mini + GPT-5 Mini + embeddings)- `numpy>=1.24.0`, `pandas>=2.0.0` — Data handling- `scikit-learn>=1.2.0` — NearestNeighbors retrieval index and metrics 7. **Install Analysis requirements:**```bashcd Analysispip install -r analysis_requirements.txtcd ..``` ## Configuration All data paths are **auto-resolved relative to the project root** — no manual path editing is needed as long as the repository structure is intact. ### API Keys (GPT and RAG experiments only) ```bash# Set your OpenAI API key in:# - Gpt_Experiments/scripts/config.py (for GPT experiments)# - RAG_Experiments/code/config.py (for RAG experiments)OPENAI_API_KEY = "your-api-key-here"``` ### GitHub Token (optional, for bot analysis only) ```bash# Set in Analysis/config.py (only needed for bot_labeling_analysis.py):GITHUB_TOKEN = "ghp_..."``` ## Usage ### Fine-tuned Experiments ```bashcd Fine_tuned_Experimentspython distilbert_final.pypython roberta_final.py``` ### GPT Experiments ```bashcd Gpt_Experiments/scriptspython gpt4o_mini_zeroshot_gridsearch.py # GPT-4o Mini zero-shotpython gpt4o_mini_fewshot_gridsearch.py # GPT-4o Mini few-shotpython gpt_5_mini_zeroshot_gridsearch.py # GPT-5 Mini zero-shotpython gpt_5_mini_fewshot_gridsearch.py # GPT-5 Mini few-shotpython gpt_5_nano_zeroshot.py # GPT-5 Nano zero-shotpython gpt_5_nano_fewshot.py # GPT-5 Nano few-shot``` ### RAG Experiments Run from the project root directory. Scripts are self-contained and use `RAG_Experiments/code/config.py`. **Step 1 — Run any RAG pipeline:**```bashpython RAG_Experiments/code/01_agentic_rag_zeroshot.pypython RAG_Experiments/code/02_agentic_rag_fewshot.pypython RAG_Experiments/code/03_adaptive_rag_zeroshot.pypython RAG_Experiments/code/04_adaptive_rag_fewshot.pypython RAG_Experiments/code/05_adaptive_rag_gpt5mini.py``` **Step 2 — Run per-label threshold tuning on saved predictions:**```bashpython RAG_Experiments/code/06_threshold_tuning.py agentic_rag_zeroshot_predictions.jsonpython RAG_Experiments/code/06_threshold_tuning.py agentic_rag_fewshot_predictions.jsonpython RAG_Experiments/code/06_threshold_tuning.py adaptive_rag_zeroshot_predictions.jsonpython RAG_Experiments/code/06_threshold_tuning.py adaptive_rag_fewshot_predictions.jsonpython RAG_Experiments/code/06_threshold_tuning.py adaptive_rag_gpt5mini_predictions.json``` **Step 3 — Run direct GPT-5 Mini baseline (no RAG, same 721-issue test set):**```bashpython RAG_Experiments/code/07_gpt5mini_fewshot_direct.pypython RAG_Experiments/code/06_threshold_tuning.py gpt5mini_fewshot_direct_predictions.json``` > **Note on runtime**: Each RAG script makes ~1–3 API calls per test issue (721 issues total). Expect 30–120 minutes per script depending on model and network latency. ### Analysis ```bashcd Analysispython ml_baselines.py # Classical ML baselines (LR, SVM)python statisticalanalysis.py # McNemar's significance testspython quantum_term_analysis.py # Quantum terminology analysispython bot_labeling_analysis.py # Bot labeling analysis (requires GitHub token)``` ## Input Data Format The experiments expect a JSON file with GitHub issues in the following format: ```json[ { "ID": "issue-123", "Title": "Fix barrier label position when bits are reversed", "Body": "Issue description text here...", "Labels": ["mod: visualization", "bug"] }]``` ### Dataset Details- **Source**: Qiskit GitHub repository issues- **Total**: 2,415 issues across 12 quantum-specific categories- **Labels**: `mod: algorithms`, `mod: circuit`, `mod: opflow`, `mod: primitives`, `mod: pulse`, `mod: qasm2`, `mod: qasm3`, `mod: qpy`, `mod: quantum info`, `mod: transpiler`, `mod: visualization`, `qamp`- **Label type**: Multi-label (issues can have multiple labels)- **Few-shot examples**: 13 curated examples excluded from all evaluation sets ## Output and Results ### Fine-tuned Models- `finetuned_distilbert_results/DistilBERT_Results/`- `finetuned_roberta_results/RoBERTa_Results/` ### GPT Models- `gpt_4o_mini_results/` — GPT-4o Mini results (grid search)- `gpt_5_mini_results/` — GPT-5 Mini results- `gpt_5_nano_results/` — GPT-5 Nano results ### RAG Experiments- `RAG_Experiment