Does Context Size Matter? Context Structure and Integration Effort in Agent Pull Requests

This is the replication package for the paper: Does Context Size Matter? An Empirical Study of Agent-Authored Pull Requests All experiments are provided as Python notebooks and were run on Google Colaboratory. Requires: Huggingface API key OpenAI API key Connection to Google Drive with required files (to run the categorization and evaluations) GitHub token (to fetch commit dates, although we provide the extracted data) The zip package contains the following files: Data exploration: data_exploration: AIDev-pop data exploration.ipynb: general data exploration of the dataset, fix PR types. For RQ1: first_commits (RQ1): AIDev-pop single-bot-commits.ipynb For RQ2: commits_and_comments_past1st: AIDev-Fix Review Commits-Comments.ipynb: notebook to explore commits and comments commits_data: AIDev-pop fetch commit dates.ipynb: fetch missing commit dates associated with commit details for agent commits. commit_datetime_cache.csv: extracted commit dates with PR ids. failed_commits_humans.csv: extracted commits on rejected agent PRs by human users success_commits_humans.csv: extracted commits on accepted agent PRs by human users comments_data: pr_comments_failed_full.csv: extraction of comments on rejected PRs pr_comments_success_full.csv: extraction of comments on accepted PRs llm_categorization: AIDev comment categorization.ipynb: automated labelling of human review comments by GPT4.1 comments-eval.ipynb: results analysis, automated + manual labeling of comments categorize-commits.ipynb: automated labeling of human commits by GPT4.1 commits-eval.ipynb: results analysis, automated + manual labeling of commits success_commits_llm.csv: LLM categorization of commits on accepted PRs success_comments_llm.csv: LLM categorization of comments on accepted PRs manual_categorization: pr_comments_failed_sample_w_cat.csv: 50 sampled comments for manual evaluation, rejected PRs pr_comments_success_sample_w_cat.csv: 50sampled comments for manual evaluation, accepted PRs sampled_commits_fail_w_cat.csv: 50sampled commits for manual evaluation, rejected PRs sampled_commits_success_w_cat.csv: 50 sampled commits for manual evaluation, accepted PRs