On the Footprints of Reviewer Bots Feedback on Agentic Pull Requests in OSS GitHub Repositories

DescriptionThis replication kit includes: review_bot.py: Main analysis script for evaluating comment quality, tag distributions, and computing agreement scores. review_comments_evaluated_with_body.csv: LLM-evaluated bot comments with relevance, clarity, conciseness scores. Manual_200_sample_reviews.csv and GPT_200_sample_reviews.csv: Annotated samples for inter-rater agreement. bot_comment_scores.csv: Pre-evaluated quality scores for bot-generated review comments. requirements.txt: Dependencies for replicating results. Categorization Framework: Consists of the prompts and results of categories The script generates all quantitative results and plots presented in the study. See README for usage instructions.