Replication package for Tag-Aware High-Priority Issue Classification

Overview This replication package provides the necessary artifacts, datasets, and source code to fully reproduce the findings of the study "Tag-Aware High-Priority Issue Classification." The study rigorously evaluates the performance and robustness of various language models (both general-purpose LLMs and fine-tuned encoders) in identifying high-priority software issues. A central focus of this research is quantifying how models behave under input representation shift—specifically, how the addition or removal of predicted structural metadata tags (such as affected components or technical debt categories) impacts classification accuracy, robustness, and reliability in automated triage pipelines. Directory Contents 📂 All_Dataset/ Structure: The data is formulated for binary classification (high-priority vs. non-priority) and is strictly partitioned into train, validation, and test splits. Crucially, the data is also stratified into semantic domains (Bugs vs. Features) to allow for domain-aware validation, as the study proves that aggregate metrics often hide severe, domain-specific failures. 📂 Notebooks with all training code/ Context: Contains the scripts and Jupyter notebooks used to execute the end-to-end experimental workflow. Usage: This code facilitates the comparative benchmarking of generalist zero-shot Large Language Models (such as GPT-3.5 Turbo, Phi-4-14b, and GPT-5-nano) against the specialist, fine-tuned DistilRoberta encoder. It covers both the text-only and text+tags input configurations. 📊 Human Baseline Evaluation Files: student1.xlsx, student2.xlsx, student3.xlsx, allstudents.xlsx, student_evaluation_analysis.py Context: Automated metrics often lack nuance regarding the inherent human difficulty of issue triage. To establish a human baseline, three master's students with software engineering experience manually tagged a subset of issues. The student_evaluation_analysis.py script processes this data to measure human accuracy, precision, recall, F1-score, and inter-rater reliability (Cohen's Kappa) under two strict conditions: raw text versus text augmented with tags. 📦 Trained Models Files: Priority_with_tag_model.zip, Priority_without_tag_model.zip Context: The study demonstrates that a model's robustness is heavily dependent on its training conditions. Two separate DistilRoberta instances were trained to evaluate this phenomenon. Usage: * Priority_without_tag_model.zip: The baseline DistilRoberta model trained using only the raw textual content (concatenated title and description). Priority_with_tag_model.zip: The tag-aware DistilRoberta model trained on issue text augmented with structured metadata tags (Technical Debt and quality attributes) predicted by the upstream TDsuites framework.