Reassessing Dataset Quality and Fine-Tuning Practices for Vulnerability Detection on PrimeVul dataset (dataset and script)

PrimeVul Vulnerability Detection Framework: A Modular Training and Evaluation System for Code Vulnerability Classification This repository contains a modular software framework for training and evaluating large language models on code vulnerability detection tasks using the PrimeVul dataset. The framework supports zero-shot evaluation and fine-tuning (LoRA and full fine-tuning) of transformer-based models for binary classification of code functions as vulnerable or secure. Key Features: Zero-shot evaluation using log-probability scoring of verbalizer tokens Parameter-efficient fine-tuning with LoRA (Low-Rank Adaptation) using QLoRA Full model fine-tuning support Early stopping to prevent overfitting Multi-run experimental support with statistical summary generation Comprehensive metrics: accuracy, precision, recall, F1-score, and confusion matrix components Per-sample prediction export for detailed analysis Architecture: The codebase is organized as a modern Python package with modular components: src/data/: Data loading utilities for JSONL-formatted code samples src/models/: Model and tokenizer configuration utilities src/training/: Training functions for LoRA and full fine-tuning src/evaluation/: Evaluation functions with log-probability scoring src/utils/: Prompt templates, data collators, and statistics utilities Requirements: Python 3.8+ PyTorch (nightly build with CUDA 12.8 recommended) Transformers, PEFT, Datasets libraries SentencePiece and Protobuf for tokenizer support Usage: The framework provides a command-line interface for both zero-shot evaluation and fine-tuning workflows. All parameters are configurable via command-line arguments, including batch sizes, learning rates, context lengths, attention implementations, and more.