Scripts, Software and Dataset for "Benchmarking LLM Commit Message Generation through a Developer-centric Pairwise Preference Framework"

# Benchmarking LLM Commit Message Generation through a Developer-centric Pairwise Preference Framework This repository is the replication package of the paper "Benchmarking LLM Commit Message Generation through a Developer-centric Pairwise Preference Framework". The purpose of this repository is to provide the code, data and scripts required to reproduce the experiments and metrics described in the paper. ## Repository contents - `automated_metrics/` — Scripts for automatic metric calculation and statistics (e.g., METEOR). - `diff_mining/` — Diff data, samples and mining scripts used in experiments. - `dataset_experiment/` — Dataset of diffs evaluated during our commit message preference analysis. - `elo/` — Scripts and results related to preference ranking analysis, numbered scripts should be executed in ascending order. - `CoMEHT/` — The back and frontend implementations of our Commit Message Evaluation Helper Tool for replication purposes.